You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dabla (via GitHub)" <gi...@apache.org> on 2026/10/04 17:55:46 UTC

[PR] Add native async execution to AgentOperator and @task.agent [airflow]

dabla opened a new pull request, #74201:
URL: https://github.com/apache/airflow/pull/74201

   `@task.agent` cannot be used with an `async def` function today. The decorator calls the function without awaiting it, so the task fails in `validate_prompt` on the coroutine it gets back, while the operator already reports `is_async` through `DecoratedOperator`. An agent task therefore cannot await async hooks to build its prompt, and cannot share the event loop of an async task with other work, although the pydantic-ai agent behind it is async-native.
   
   This makes `AgentOperator` a `BaseAsyncOperator`, the way `PythonOperator` is one:
   
   - `@task.agent` on an `async def` function awaits it for the prompt and runs the agent on the task's event loop. A regular function keeps the synchronous path unchanged. `AgentOperator.is_async` is `False`; on the decorator it follows the function.
   - `AgentOperator.aexecute()` makes no blocking call to the supervisor on the path of a successful run: `PydanticAIHook.aget_hook()` fetches the connection once and primes the hook with it, the agent is built with `acreate_agent()` and run with `await agent.run()` (`CancellableAgentRunMixin.run_agent_async`, same cancellation token as the sync run), and the usage budget (`aload` / `asave` / `aclear`), the tool approval transcript and the XComs go through their async accessors.
   - The two rare branches reuse the synchronous code from a worker thread: the usage report of a failed run and the pause for a tool approval. Resuming after an approval stays synchronous.
   - The template fields are rendered in a worker thread, since a template can read a Variable or a Connection with a blocking call.
   - `durable=True` and `enable_hitl_review=True` do blocking I/O during the run (the durable storage inside the model and tool calls, the review poll loop) and are rejected with an `async def` function, at parse time on the decorator. They can follow in a later PR.
   - `axcom_push` exists since Airflow 3.3.0; on 3.2 the XComs are pushed with `xcom_push` from a worker thread.
   
   Depends on #74150 (`aget_conn` / `acreate_agent` on `PydanticAIHook`): this branch is built on top of it, so the diff shows its commits until it is merged.
   
   **What it changes for performance.** One agent run is not faster, its duration is the model's. The gain is in running many of them in one task process. Measured with pydantic-ai alone and a fake model that sleeps 1 s, one run per cell: a thread pool calling `run_sync` versus one event loop awaiting `run`.
   
   | Concurrent runs | Thread pool + `run_sync` | One loop + `run` | CPU, threads | CPU, loop |
   |---|---|---|---|---|
   | 8 | 1.06 s | 1.03 s | 0.07 s | 0.02 s |
   | 32 | 1.26 s | 1.10 s | 0.30 s | 0.09 s |
   | 64 | 1.41 s | 1.19 s | 0.51 s | 0.18 s |
   | 256 | 2.97 s | 1.82 s | 2.33 s | 0.78 s |
   
   **Tests.** New tests cover `run_agent_async` (token held and cleared, `on_kill` cancelling an awaited run), `aget_hook`, the async usage budget methods, `aexecute` with mocks and with a real `FunctionModel` run where every blocking state store call and `xcom_push` fails the test, the failed-run and tool-approval branches, the rejected features, the decorator with an `async def` function through both `aexecute` and `execute`, rendering off the event loop, and eight runs overlapping on one loop.
   
   Run locally: the whole `providers/common/ai/tests/unit` suite (3104 passed, 13 skipped), `mypy` on the five changed source files, and `prek` on the changed files. A backport of the same design to common.ai 0.9 also ran on a Celery worker with Airflow 3.3.2: one async `@task.agent` task, and eight agent runs sharing one event loop inside a single task, against an Azure OpenAI model, without a `DeadlockImminentError`.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Fable 5.1)
   
   Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "dabla (via GitHub) via commits" <co...@airflow.apache.org>.
dabla commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4552269372


##########
providers/common/ai/src/airflow/providers/common/ai/decorators/agent.py:
##########
@@ -106,6 +117,19 @@ def execute(self, context: Context) -> Any:
         self.prompt = self.render_template(self.prompt, context)
         return AgentOperator.execute(self, context)
 
+    async def aexecute(self, context: Context) -> Any:
+        context_merge(context, self.op_kwargs)
+        kwargs = determine_kwargs(self.python_callable, self.op_args, context)
+
+        self.prompt = await self.python_callable(*self.op_args, **kwargs)
+
+        validate_prompt(self.prompt, decorator_name="@task.agent")
+
+        # Rendering a template can read a Variable or a Connection with a blocking call to
+        # the supervisor, which must not happen on the event loop.
+        await asyncio.to_thread(self.render_template_fields, context)

Review Comment:
   You're right, and the merge of main made it worse: `execute` got #76244 and `aexecute` kept the old line. Fixed in 1fdc7937f4: `aexecute` now merges `call_arguments(...)` into the context and renders only the returned prompt, with `asyncio.to_thread(self.render_template, self.prompt, context)`. Two tests pin it, the twin of `test_execute_renders_only_the_returned_prompt` and one that calls the async function with a positional and a keyword argument.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1085,16 +1134,69 @@ def execute(self, context: Context) -> Any:
                 # Without a history to inherit one from, pydantic-ai starts a new conversation
                 # on every attempt; keep one for the durable run, for the same reason.
                 run_kwargs["conversation_id"] = run_kwargs["run_id"]
+        return run_kwargs
+
+    def _reject_unsupported_on_async_path(self) -> None:
+        """Refuse the features that block the event loop: both do synchronous I/O during the run."""
+        if self.durable or self.enable_hitl_review:
+            raise ValueError(
+                f"{type(self).__name__}: durable=True and enable_hitl_review=True are not supported "
+                "with an async callable. Use a synchronous callable for them."
+            )
+
+    async def aexecute(self, context: Context) -> Any:
+        """
+        Run the agent on the worker's event loop.
+
+        The async version of :meth:`execute`: the agent run is awaited on the event loop, and the
+        bookkeeping of a successful run (usage budget, run metadata, message history) makes no
+        blocking call to the supervisor on it. What has no async counterpart (the connection
+        lookup, the task state store before Airflow 3.4, the usage report of a failed run, the
+        pause for a tool approval) runs the synchronous code in a worker thread.
+        """
+        self._reject_unsupported_on_async_path()
+        usage_limits = coerce_usage_limits(self.usage_limits)
+
+        if self._supports_tool_approval() and (store := context.get("task_state_store")) is not None:
+            await self._adelete_approval_transcript(store)
+
+        ti = context["task_instance"]
+        self._durable_journal = None
+        self._usage_budget = self._build_usage_budget(context, usage_limits, ti=ti)
+        self._run_usage = await self._usage_budget.aload() if self._usage_budget else RunUsage()
+        run_usage: RunUsage = self._run_usage
+        self._run_usage_base = copy_run_usage(run_usage)
+
+        agent = await self._abuild_agent()
+        run_kwargs = self._run_kwargs(agent, ti, usage_limits)
 
-        # A killed run raises RunCancelled (see run_agent_sync), which propagates to fail the
-        # task. The journal cleanup in _complete_run is skipped on the raise, keeping it for the retry.
         try:
-            result, attempt_usage = self._run_and_report_on_failure(context, agent, run_usage, run_kwargs)
-        finally:
-            # Also on a raise: the failed attempt is the one Airflow retries.
-            if self._durable_journal is not None:
-                self._durable_journal.log_summary(self.log)
-        return self._complete_run(context, result, attempt_usage=attempt_usage)
+            result, attempt_usage = await self._arun_tracked(
+                agent, self.prompt, run_usage=run_usage, **run_kwargs
+            )
+        except BaseException:
+            await asyncio.to_thread(self._report_failed_run, context, run_usage, run_id=run_kwargs["run_id"])
+            raise
+        return await self._acomplete_run(context, result, attempt_usage=attempt_usage)
+
+    async def _acomplete_run(self, context: Context, result: Any, *, attempt_usage: RunUsage) -> Any:
+        """Async version of :meth:`_complete_run`, without the features :meth:`aexecute` refuses."""
+        log_run_summary(self.log, result, usage=attempt_usage)
+        if isinstance(result.output, DeferredToolRequests):

Review Comment:
   Yes, same guard now, in 4889639dd8: `_acomplete_run` catches `BaseException` around the pause, runs `_report_failed_run` in a worker thread unless the error is `TaskAwaitingInput`, and re-raises. The new test ends a run on `UnsupportedToolDeferralError` and checks the `run_id` and `usage` XComs.



##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4161,3 +4170,295 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    def test_operator_is_synchronous_unless_a_subclass_says_otherwise(self):
+        assert AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm").is_async is False
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        pushed = [c.kwargs["key"] for c in context["task_instance"].axcom_push.await_args_list]
+        assert pushed == ["run_id", "usage", MODEL_NAME_XCOM_KEY, "message_history"]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        context["task_instance"].axcom_push.assert_not_awaited()
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_retry_starts_from_the_usage_of_earlier_tries(self, mock_hook_cls, make_mock_run_result):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+        store.data[USAGE_BUDGET_KEY] = TaskStateStoreUsageBudget(
+            store, max_tries=context["task_instance"].max_tries
+        )._record(RunUsage(requests=3))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        await op.aexecute(context)
+
+        assert mock_agent.run.await_args.kwargs["usage"].requests == 3
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_approval_transcript_of_an_earlier_try_is_deleted(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        store.data[_TOOL_APPROVAL_TRANSCRIPT_KEY] = "[]"
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        await op.aexecute(_make_async_context(task_state_store=store))
+
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_output_is_returned_as_a_dict(self, mock_hook_cls, make_mock_run_result):
+        class Answer(BaseModel):
+            text: str
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(Answer(text="done")))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=Answer, serialize_output=True
+        )
+
+        assert await op.aexecute(_make_async_context()) == {"text": "done"}
+
+    @requires_typed_xcom
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_the_worker_does_not_register_is_returned_as_json(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        """The async path converts the output the way ``execute`` does: a model inside a list output is
+        dumped, since the worker registers no class for an output marker."""
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result([Summary(text="ok")]))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=ToolOutput(list[Summary])
+        )
+
+        assert await op.aexecute(_make_async_context()) == [{"text": "ok", "score": 0.0}]
+
+    @pytest.mark.parametrize(
+        ("kwargs", "emits"),
+        [
+            pytest.param({"message_history": []}, True, id="seeded"),
+            pytest.param({"emit_message_history": True}, True, id="opt-in"),
+            pytest.param({"message_history": [], "emit_message_history": False}, False, id="opt-out"),
+        ],
+    )
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_transcript_push_follows_emit_message_history(
+        self, mock_hook_cls, kwargs, emits, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("ok"))
+        mock_agent.run.return_value.all_messages.return_value = _sample_history()
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **kwargs)
+        await op.aexecute(context)
+
+        pushes = {
+            c.kwargs["key"]: c.kwargs["value"] for c in context["task_instance"].axcom_push.await_args_list

Review Comment:
   Fixed in 9caace3c3a, both ways: the tests that read the pushes now collect them from the mock the running version uses, and `_axcom_push` has its own test with the 3.3 flag patched on and off. The off case also fails if `xcom_push` runs on the event loop, so the fallback branch is covered on every version.



##########
providers/common/ai/docs/operators/agent.rst:
##########
@@ -83,6 +83,30 @@ the prompt string; all other parameters are passed to the operator.
     :start-after: [START howto_decorator_agent]
     :end-before: [END howto_decorator_agent]
 
+.. _howto/operator:agent-async:
+
+Async callables
+^^^^^^^^^^^^^^^
+
+On Airflow 3.2+ the decorated function can be an ``async def``. It is awaited for the prompt, and
+the agent run is then awaited on the task's event loop. The function can await async hooks. What
+has no async counterpart, such as the connection lookup, runs in a worker thread, so nothing blocks
+the event loop.
+
+.. code-block:: python
+
+    @task.agent(llm_conn_id="pydanticai_default", system_prompt="You are a support analyst.")
+    async def summarize_ticket(ticket_id: str) -> str:
+        ticket = await fetch_ticket(ticket_id)
+        return f"Summarize this ticket: {ticket}"
+
+A single run is not faster this way, as its duration is the model's. Tools that call blocking hooks,
+such as ``SQLToolset`` and ``HookToolset``, run in a worker thread with either kind of function.
+
+``durable=True`` and ``enable_hitl_review=True`` do blocking I/O during the run and are not
+supported with an ``async def`` function: the Dag fails to parse with a ``ValueError``. Use a regular

Review Comment:
   Good catch. I went for the docs in 18c8c6148e: the page now says a mapped task is only created when it runs, so it fails there with the same `ValueError`. A parse-time check on the mapped path would have to live in the Task SDK's `_TaskDecorator.expand()`, since the provider's operator is not built at that point, and I'd rather not add a core change to this PR. Happy to look at it as a follow-up if you think it is worth it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "kaxil (via GitHub) via commits" <co...@airflow.apache.org>.
kaxil commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4483605168


##########
providers/common/ai/src/airflow/providers/common/ai/decorators/agent.py:
##########
@@ -106,6 +117,19 @@ def execute(self, context: Context) -> Any:
         self.prompt = self.render_template(self.prompt, context)
         return AgentOperator.execute(self, context)
 
+    async def aexecute(self, context: Context) -> Any:
+        context_merge(context, self.op_kwargs)
+        kwargs = determine_kwargs(self.python_callable, self.op_args, context)
+
+        self.prompt = await self.python_callable(*self.op_args, **kwargs)
+
+        validate_prompt(self.prompt, decorator_name="@task.agent")
+
+        # Rendering a template can read a Variable or a Connection with a blocking call to
+        # the supervisor, which must not happen on the event loop.
+        await asyncio.to_thread(self.render_template_fields, context)

Review Comment:
   This renders every template field a second time rather than only the prompt, and the context never gets the positional arguments. So the docstring's own example, `analyze("x")` returning `"Answer: {{ question }}"`, fails here with `UndefinedError` while the sync path renders it. A `message_history` or `op_kwargs` value from an upstream XCom that contains `{{` also gets evaluated as Jinja on this second pass. Could this mirror `execute`: merge `call_arguments(...)` into the context, then `self.prompt = await asyncio.to_thread(self.render_template, self.prompt, context)`? A test that calls the async function with a positional argument would pin it.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1029,6 +1060,9 @@ def _run_and_report_on_failure(
             raise
 
     def execute(self, context: Context) -> Any:
+        if self.is_async:

Review Comment:
   Nothing in-tree reaches this branch: `AgentOperator.is_async` is always `False`, and the decorator dispatches in its own `execute`. Could `BaseAsyncOperator` go on `_AgentDecoratedOperator` only, so the public operator's base classes don't change? Or is an async path for the plain operator planned?



##########
providers/common/ai/docs/operators/agent.rst:
##########
@@ -83,6 +83,30 @@ the prompt string; all other parameters are passed to the operator.
     :start-after: [START howto_decorator_agent]
     :end-before: [END howto_decorator_agent]
 
+.. _howto/operator:agent-async:
+
+Async callables
+^^^^^^^^^^^^^^^
+
+On Airflow 3.2+ the decorated function can be an ``async def``. It is awaited for the prompt, and
+the agent run is then awaited on the task's event loop. The function can await async hooks. What
+has no async counterpart, such as the connection lookup, runs in a worker thread, so nothing blocks

Review Comment:
   Not quite with `AgentSkillsToolset`: its `__aexit__` still calls `xcom_pull`/`xcom_push` synchronously on the loop thread. Could this sentence be narrowed to the operator's own calls, or that push moved to a thread?



##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4161,3 +4170,295 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    def test_operator_is_synchronous_unless_a_subclass_says_otherwise(self):
+        assert AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm").is_async is False
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        pushed = [c.kwargs["key"] for c in context["task_instance"].axcom_push.await_args_list]
+        assert pushed == ["run_id", "usage", MODEL_NAME_XCOM_KEY, "message_history"]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        context["task_instance"].axcom_push.assert_not_awaited()
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_retry_starts_from_the_usage_of_earlier_tries(self, mock_hook_cls, make_mock_run_result):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+        store.data[USAGE_BUDGET_KEY] = TaskStateStoreUsageBudget(
+            store, max_tries=context["task_instance"].max_tries
+        )._record(RunUsage(requests=3))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        await op.aexecute(context)
+
+        assert mock_agent.run.await_args.kwargs["usage"].requests == 3
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_approval_transcript_of_an_earlier_try_is_deleted(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        store.data[_TOOL_APPROVAL_TRANSCRIPT_KEY] = "[]"
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        await op.aexecute(_make_async_context(task_state_store=store))
+
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_output_is_returned_as_a_dict(self, mock_hook_cls, make_mock_run_result):
+        class Answer(BaseModel):
+            text: str
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(Answer(text="done")))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=Answer, serialize_output=True
+        )
+
+        assert await op.aexecute(_make_async_context()) == {"text": "done"}
+
+    @requires_typed_xcom
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_the_worker_does_not_register_is_returned_as_json(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        """The async path converts the output the way ``execute`` does: a model inside a list output is
+        dumped, since the worker registers no class for an output marker."""
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result([Summary(text="ok")]))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=ToolOutput(list[Summary])
+        )
+
+        assert await op.aexecute(_make_async_context()) == [{"text": "ok", "score": 0.0}]
+
+    @pytest.mark.parametrize(
+        ("kwargs", "emits"),
+        [
+            pytest.param({"message_history": []}, True, id="seeded"),
+            pytest.param({"emit_message_history": True}, True, id="opt-in"),
+            pytest.param({"message_history": [], "emit_message_history": False}, False, id="opt-out"),
+        ],
+    )
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_transcript_push_follows_emit_message_history(
+        self, mock_hook_cls, kwargs, emits, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("ok"))
+        mock_agent.run.return_value.all_messages.return_value = _sample_history()
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **kwargs)
+        await op.aexecute(context)
+
+        pushes = {
+            c.kwargs["key"]: c.kwargs["value"] for c in context["task_instance"].axcom_push.await_args_list

Review Comment:
   This only reads `axcom_push`, but on 3.2 `_axcom_push` falls back to `xcom_push` in a thread, so `[seeded]` and `[opt-in]` fail there. That is the red Compat 3.2.2 job. Collecting from both mocks would fix it, or skip this below 3.3 and add a test with the 3.3 flag patched off, which would also cover the fallback branch (it has no test yet).



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1085,16 +1134,69 @@ def execute(self, context: Context) -> Any:
                 # Without a history to inherit one from, pydantic-ai starts a new conversation
                 # on every attempt; keep one for the durable run, for the same reason.
                 run_kwargs["conversation_id"] = run_kwargs["run_id"]
+        return run_kwargs
+
+    def _reject_unsupported_on_async_path(self) -> None:
+        """Refuse the features that block the event loop: both do synchronous I/O during the run."""
+        if self.durable or self.enable_hitl_review:
+            raise ValueError(
+                f"{type(self).__name__}: durable=True and enable_hitl_review=True are not supported "
+                "with an async callable. Use a synchronous callable for them."
+            )
+
+    async def aexecute(self, context: Context) -> Any:
+        """
+        Run the agent on the worker's event loop.
+
+        The async version of :meth:`execute`: the agent run is awaited on the event loop, and the
+        bookkeeping of a successful run (usage budget, run metadata, message history) makes no
+        blocking call to the supervisor on it. What has no async counterpart (the connection
+        lookup, the task state store before Airflow 3.4, the usage report of a failed run, the
+        pause for a tool approval) runs the synchronous code in a worker thread.
+        """
+        self._reject_unsupported_on_async_path()
+        usage_limits = coerce_usage_limits(self.usage_limits)
+
+        if self._supports_tool_approval() and (store := context.get("task_state_store")) is not None:
+            await self._adelete_approval_transcript(store)
+
+        ti = context["task_instance"]
+        self._durable_journal = None
+        self._usage_budget = self._build_usage_budget(context, usage_limits, ti=ti)
+        self._run_usage = await self._usage_budget.aload() if self._usage_budget else RunUsage()
+        run_usage: RunUsage = self._run_usage
+        self._run_usage_base = copy_run_usage(run_usage)
+
+        agent = await self._abuild_agent()
+        run_kwargs = self._run_kwargs(agent, ti, usage_limits)
 
-        # A killed run raises RunCancelled (see run_agent_sync), which propagates to fail the
-        # task. The journal cleanup in _complete_run is skipped on the raise, keeping it for the retry.
         try:
-            result, attempt_usage = self._run_and_report_on_failure(context, agent, run_usage, run_kwargs)
-        finally:
-            # Also on a raise: the failed attempt is the one Airflow retries.
-            if self._durable_journal is not None:
-                self._durable_journal.log_summary(self.log)
-        return self._complete_run(context, result, attempt_usage=attempt_usage)
+            result, attempt_usage = await self._arun_tracked(
+                agent, self.prompt, run_usage=run_usage, **run_kwargs
+            )
+        except BaseException:
+            await asyncio.to_thread(self._report_failed_run, context, run_usage, run_id=run_kwargs["run_id"])
+            raise
+        return await self._acomplete_run(context, result, attempt_usage=attempt_usage)
+
+    async def _acomplete_run(self, context: Context, result: Any, *, attempt_usage: RunUsage) -> Any:
+        """Async version of :meth:`_complete_run`, without the features :meth:`aexecute` refuses."""
+        log_run_summary(self.log, result, usage=attempt_usage)
+        if isinstance(result.output, DeferredToolRequests):

Review Comment:
   The sync `_complete_run` calls `_report_failed_run` when the approval pause is refused (`UnsupportedToolDeferralError`, `ToolApprovalAlreadyRequestedError`), but here the error goes straight up. An async attempt that ends this way pushes no `run_id`/`usage` XCom, which the class docstring promises for a failed attempt. Should this get the same `except BaseException` guard, with `_report_failed_run` run in a thread?



##########
providers/common/ai/docs/operators/agent.rst:
##########
@@ -83,6 +83,30 @@ the prompt string; all other parameters are passed to the operator.
     :start-after: [START howto_decorator_agent]
     :end-before: [END howto_decorator_agent]
 
+.. _howto/operator:agent-async:
+
+Async callables
+^^^^^^^^^^^^^^^
+
+On Airflow 3.2+ the decorated function can be an ``async def``. It is awaited for the prompt, and
+the agent run is then awaited on the task's event loop. The function can await async hooks. What
+has no async counterpart, such as the connection lookup, runs in a worker thread, so nothing blocks
+the event loop.
+
+.. code-block:: python
+
+    @task.agent(llm_conn_id="pydanticai_default", system_prompt="You are a support analyst.")
+    async def summarize_ticket(ticket_id: str) -> str:
+        ticket = await fetch_ticket(ticket_id)
+        return f"Summarize this ticket: {ticket}"
+
+A single run is not faster this way, as its duration is the model's. Tools that call blocking hooks,
+such as ``SQLToolset`` and ``HookToolset``, run in a worker thread with either kind of function.
+
+``durable=True`` and ``enable_hitl_review=True`` do blocking I/O during the run and are not
+supported with an ``async def`` function: the Dag fails to parse with a ``ValueError``. Use a regular

Review Comment:
   A mapped task still parses: the check is in `__init__`, which `.expand()` doesn't call until the task runs. Maybe say the task fails with a `ValueError`, or add the check on the mapped path too?



##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4161,3 +4170,295 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    def test_operator_is_synchronous_unless_a_subclass_says_otherwise(self):
+        assert AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm").is_async is False
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        pushed = [c.kwargs["key"] for c in context["task_instance"].axcom_push.await_args_list]
+        assert pushed == ["run_id", "usage", MODEL_NAME_XCOM_KEY, "message_history"]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        context["task_instance"].axcom_push.assert_not_awaited()
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_retry_starts_from_the_usage_of_earlier_tries(self, mock_hook_cls, make_mock_run_result):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+        store.data[USAGE_BUDGET_KEY] = TaskStateStoreUsageBudget(
+            store, max_tries=context["task_instance"].max_tries
+        )._record(RunUsage(requests=3))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        await op.aexecute(context)
+
+        assert mock_agent.run.await_args.kwargs["usage"].requests == 3
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_approval_transcript_of_an_earlier_try_is_deleted(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        store.data[_TOOL_APPROVAL_TRANSCRIPT_KEY] = "[]"
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        await op.aexecute(_make_async_context(task_state_store=store))
+
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_output_is_returned_as_a_dict(self, mock_hook_cls, make_mock_run_result):
+        class Answer(BaseModel):
+            text: str
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(Answer(text="done")))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=Answer, serialize_output=True
+        )
+
+        assert await op.aexecute(_make_async_context()) == {"text": "done"}
+
+    @requires_typed_xcom
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_the_worker_does_not_register_is_returned_as_json(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        """The async path converts the output the way ``execute`` does: a model inside a list output is
+        dumped, since the worker registers no class for an output marker."""
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result([Summary(text="ok")]))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=ToolOutput(list[Summary])
+        )
+
+        assert await op.aexecute(_make_async_context()) == [{"text": "ok", "score": 0.0}]
+
+    @pytest.mark.parametrize(
+        ("kwargs", "emits"),
+        [
+            pytest.param({"message_history": []}, True, id="seeded"),
+            pytest.param({"emit_message_history": True}, True, id="opt-in"),
+            pytest.param({"message_history": [], "emit_message_history": False}, False, id="opt-out"),
+        ],
+    )
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_transcript_push_follows_emit_message_history(
+        self, mock_hook_cls, kwargs, emits, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("ok"))
+        mock_agent.run.return_value.all_messages.return_value = _sample_history()
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **kwargs)
+        await op.aexecute(context)
+
+        pushes = {
+            c.kwargs["key"]: c.kwargs["value"] for c in context["task_instance"].axcom_push.await_args_list
+        }
+        assert ("message_history" in pushes) is emits
+        if emits:
+            assert len(ModelMessagesTypeAdapter.validate_json(pushes["message_history"])) == 2
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_runs_overlap_on_one_event_loop(self, mock_hook_cls):
+        async def slow(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
+            await asyncio.sleep(0.2)
+            return ModelResponse(parts=[TextPart(content="the answer")])
+
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(slow), **kw)
+        )
+        ops = [AgentOperator(task_id=f"test_{i}", prompt="run", llm_conn_id="my_llm") for i in range(8)]
+
+        loop = asyncio.get_running_loop()
+        started = loop.time()
+        results = await asyncio.gather(*(op.aexecute(_make_async_context()) for op in ops))
+
+        assert results == ["the answer"] * 8
+        # Eight runs of 0.2 s each take 1.6 s one after the other.
+        assert loop.time() - started < 1.0

Review Comment:
   This also passes if `_arun_tracked` hands `run_sync` to `asyncio.to_thread`, since eight threads overlap just as well. Recording `threading.get_ident()` in `slow` and asserting it is the loop's thread would pin the one-loop claim.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "dabla (via GitHub) via commits" <co...@airflow.apache.org>.
dabla commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4556861074


##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4351,3 +4362,342 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _pushed_xcoms(ti) -> dict[str, Any]:
+    """The XComs ``aexecute`` pushed, in order: through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    calls = ti.axcom_push.await_args_list if AIRFLOW_V_3_3_PLUS else ti.xcom_push.call_args_list
+    return {c.kwargs["key"]: c.kwargs["value"] for c in calls}
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        assert list(_pushed_xcoms(context["task_instance"])) == [
+            "run_id",
+            "usage",
+            MODEL_NAME_XCOM_KEY,
+            "message_history",
+        ]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        assert _pushed_xcoms(context["task_instance"]) == {}

Review Comment:
   Done in 36005df14d: the pause now raises a real `TaskAwaitingInput` and the test asserts `_report_failed_run` is not called, so it covers the `not _is_awaiting_input(e)` guard and no longer depends on which push mock the helper reads. It is skipped below 3.3, where the exception does not exist.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -139,6 +140,14 @@
 _RUN_USAGE_ADAPTER: TypeAdapter[RunUsage] = TypeAdapter(RunUsage)
 
 
+async def _axcom_push(ti: Any, *, key: str, value: Any) -> None:
+    if AIRFLOW_V_3_3_PLUS:
+        await ti.axcom_push(key=key, value=value)
+    else:
+        # Airflow 3.2 runs async tasks but has no ``axcom_push``: push from a worker thread.
+        await asyncio.to_thread(ti.xcom_push, key=key, value=value)

Review Comment:
   You're right, and `asend()` takes `_thread_lock` through that executor too. In 5669cfddcf the hook's one-worker executor moved to `utils/coroutines.py` as `run_off_loop`, and all of these go through it: the 3.2 XCom push, both `_report_failed_run` calls, the approval pause, the transcript delete, the prompt render, the usage budget calls before 3.4, and the skills XCom push from the last round. The hook uses the same helper, so the provider has one worker for its blocking supervisor calls. Each path has a test that runs with the default executor's only worker taken.
   
   I did not use `AirflowToolset.run_blocking`: it goes through `asyncio.to_thread` itself, so calls waiting on its lock sit in default-executor threads as well. That is on main and applies to every toolset. Putting it on the same single worker would let one slow tool call hold up the bookkeeping of every other run, so I'd look at it in a separate PR.
   
   #76545 removes the cause in the Task SDK: there `asend()` does the whole exchange on one worker thread that holds the lock, so waiting `send()` calls can no longer keep the read from running. That only helps on the Airflow version that ships it, so the provider still needs its own worker for the older ones it supports.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1081,16 +1117,76 @@ def execute(self, context: Context) -> Any:
                 # Without a history to inherit one from, pydantic-ai starts a new conversation
                 # on every attempt; keep one for the durable run, for the same reason.
                 run_kwargs["conversation_id"] = run_kwargs["run_id"]
+        return run_kwargs
+
+    def _reject_unsupported_on_async_path(self) -> None:
+        """Refuse the features that block the event loop: both do synchronous I/O during the run."""
+        if self.durable or self.enable_hitl_review:
+            raise ValueError(
+                f"{type(self).__name__}: durable=True and enable_hitl_review=True are not supported "
+                "with an async callable. Use a synchronous callable for them."
+            )
+
+    async def aexecute(self, context: Context) -> Any:
+        """
+        Run the agent on the worker's event loop.
+
+        ``@task.agent`` calls this for an ``async def`` function; the operator itself runs through
+        :meth:`execute`. The agent run is awaited on the event loop, and the bookkeeping of a successful run (usage budget, run metadata, message history) makes no

Review Comment:
   Re-wrapped in 4980616c05, as two paragraphs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "dabla (via GitHub) via commits" <co...@airflow.apache.org>.
dabla commented on PR #74201:
URL: https://github.com/apache/airflow/pull/74201#issuecomment-6836828593

   Fair question. For a single task you're right: one run is no faster, and the table only applies once several runs share a loop.
   
   The use case is agent runs inside an iterated async task (#62922): N runs share one loop in one process, which is what the numbers measure. That needs `aexecute` on the operator behind `@task.agent`, so this PR is the piece that lets the two work together once #62922 is in. The backport mentioned in the description ran eight runs that way in one task.
   
   A smaller one next to it: prompt functions with async-only inputs, such as an async hook or `Variable.aget` helpers shared with the other async tasks of the Dag. We run a daily report task like that.
   
   And `async def` under `@task.agent` currently fails in `validate_prompt` on the coroutine while the operator reports `is_async`, so it needs either this or an explicit error.
   
   `durable` and HITL are refused for now because they block during the run, not by design; they can follow once they have async paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "kaxil (via GitHub) via commits" <co...@airflow.apache.org>.
kaxil commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4556546177


##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4351,3 +4362,342 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _pushed_xcoms(ti) -> dict[str, Any]:
+    """The XComs ``aexecute`` pushed, in order: through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    calls = ti.axcom_push.await_args_list if AIRFLOW_V_3_3_PLUS else ti.xcom_push.call_args_list
+    return {c.kwargs["key"]: c.kwargs["value"] for c in calls}
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        assert list(_pushed_xcoms(context["task_instance"])) == [
+            "run_id",
+            "usage",
+            MODEL_NAME_XCOM_KEY,
+            "message_history",
+        ]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        assert _pushed_xcoms(context["task_instance"]) == {}

Review Comment:
   This is the test failing the Compat job now. `Paused` is not what `_is_awaiting_input` recognises, so `_acomplete_run` takes the failure branch and reports `run_id`/`usage` through `xcom_push`, which `_pushed_xcoms` reads when `axcom_push` is unavailable. Where `axcom_push` exists it only passes because those pushes are sync and the helper reads the async one. Could this raise the real `TaskAwaitingInput` and assert `_report_failed_run` is not called? That would also cover the `not _is_awaiting_input(e)` guard, which no test hits now.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -139,6 +140,14 @@
 _RUN_USAGE_ADAPTER: TypeAdapter[RunUsage] = TypeAdapter(RunUsage)
 
 
+async def _axcom_push(ti: Any, *, key: str, value: Any) -> None:
+    if AIRFLOW_V_3_3_PLUS:
+        await ti.axcom_push(key=key, value=value)
+    else:
+        # Airflow 3.2 runs async tasks but has no ``axcom_push``: push from a worker thread.
+        await asyncio.to_thread(ti.xcom_push, key=key, value=value)

Review Comment:
   `asyncio.to_thread` uses the default executor, and `CommsDecoder.asend()` reads its reply on that same executor while it holds `_thread_lock`. A sync `send()` from a worker thread just blocks on the lock. So if enough of these calls pile up, they take every slot and the read never gets a thread. Same goes for `_report_failed_run` (1161, 1177), the transcript delete and the prompt render in the decorator.
   
   `hooks/pydantic_ai.py` already hit this and runs its lookups on a one-worker executor of its own. Route these through that, or `AirflowToolset.run_blocking`?



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1081,16 +1117,76 @@ def execute(self, context: Context) -> Any:
                 # Without a history to inherit one from, pydantic-ai starts a new conversation
                 # on every attempt; keep one for the durable run, for the same reason.
                 run_kwargs["conversation_id"] = run_kwargs["run_id"]
+        return run_kwargs
+
+    def _reject_unsupported_on_async_path(self) -> None:
+        """Refuse the features that block the event loop: both do synchronous I/O during the run."""
+        if self.durable or self.enable_hitl_review:
+            raise ValueError(
+                f"{type(self).__name__}: durable=True and enable_hitl_review=True are not supported "
+                "with an async callable. Use a synchronous callable for them."
+            )
+
+    async def aexecute(self, context: Context) -> Any:
+        """
+        Run the agent on the worker's event loop.
+
+        ``@task.agent`` calls this for an ``async def`` function; the operator itself runs through
+        :meth:`execute`. The agent run is awaited on the event loop, and the bookkeeping of a successful run (usage budget, run metadata, message history) makes no

Review Comment:
   The edit joined two lines here, so this one runs to 160 columns and the next sentence starts mid-paragraph. Could it be re-wrapped?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "dabla (via GitHub) via commits" <co...@airflow.apache.org>.
dabla commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4552272104


##########
providers/common/ai/docs/operators/agent.rst:
##########
@@ -83,6 +83,30 @@ the prompt string; all other parameters are passed to the operator.
     :start-after: [START howto_decorator_agent]
     :end-before: [END howto_decorator_agent]
 
+.. _howto/operator:agent-async:
+
+Async callables
+^^^^^^^^^^^^^^^
+
+On Airflow 3.2+ the decorated function can be an ``async def``. It is awaited for the prompt, and
+the agent run is then awaited on the task's event loop. The function can await async hooks. What
+has no async counterpart, such as the connection lookup, runs in a worker thread, so nothing blocks

Review Comment:
   I moved the push rather than narrow the sentence, in 575138018d: `__aexit__` now runs `_push_skills_xcom` through `AirflowToolset.run_blocking`, like the blocking calls of the other toolsets. It matters beyond the docs: on a loop shared with other runs the sync call could hit `DeadlockImminentError`, which the `except Exception` in there would not catch. A test awaits a run with the toolset and checks both XCom calls happen off the loop thread.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -1029,6 +1060,9 @@ def _run_and_report_on_failure(
             raise
 
     def execute(self, context: Context) -> Any:
+        if self.is_async:

Review Comment:
   Agreed, the branch was there for an opt-in on the plain operator that this PR does not add ([YAGNI](https://martinfowler.com/bliki/Yagni.html)). In f76e99d7bb `BaseAsyncOperator` is a base of `_AgentDecoratedOperator` only, and `AgentOperator` has the same base classes as on main, without the `is_async` property and the dead branch in `execute`. `aexecute` stays on `AgentOperator` as the body the decorator awaits. If an async opt-in for the plain operator comes later, it brings the base class with it.



##########
providers/common/ai/tests/unit/common/ai/operators/test_agent.py:
##########
@@ -4161,3 +4170,295 @@ def test_a_toolset_that_masks_its_own_output_is_not_wrapped_again(self):
         op._build_agent()
 
         assert hook.create_agent.call_args.kwargs["toolsets"] == [sql]
+
+
+def _refused_on_the_event_loop(call):
+    """Wrap a blocking call so it fails the test when it runs on the event loop."""
+
+    def guarded(*args, **kwargs):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return call(*args, **kwargs)
+        raise AssertionError("blocking call to the supervisor on the event loop")
+
+    return guarded
+
+
+def _make_async_task_state_store_accessor(*, async_methods: bool = True):
+    """
+    A task state store backed by ``accessor.data``, whose blocking methods fail on the event loop.
+
+    Every write and delete, async or blocking, is recorded in ``accessor.ops``. With
+    ``async_methods=False`` the store has no async methods, as before Airflow 3.4, and the operator
+    has to call the blocking ones from a worker thread.
+    """
+    from airflow.sdk.execution_time.context import TaskStateStoreAccessor
+
+    accessor = MagicMock(spec=TaskStateStoreAccessor)
+    data: dict[str, Any] = {}
+    ops: list[tuple[str, str]] = []
+    accessor.data = data
+    accessor.ops = ops
+
+    def get(key, default=None):
+        return data.get(key, default)
+
+    def set_(key, value, retention=None):
+        ops.append(("set", key))
+        data[key] = value
+
+    def delete(key):
+        ops.append(("delete", key))
+        data.pop(key, None)
+
+    accessor.get.side_effect = _refused_on_the_event_loop(get)
+    accessor.set.side_effect = _refused_on_the_event_loop(set_)
+    accessor.delete.side_effect = _refused_on_the_event_loop(delete)
+    if async_methods:
+        accessor.aget = AsyncMock(side_effect=get)
+        accessor.aset = AsyncMock(side_effect=set_)
+        accessor.adelete = AsyncMock(side_effect=delete)
+    else:
+        del accessor.aget, accessor.aset, accessor.adelete
+    return accessor
+
+
+def _make_async_context(task_state_store=None):
+    """A context for ``aexecute``: XComs go through ``axcom_push`` on Airflow 3.3+, ``xcom_push`` on 3.2."""
+    ti = _make_ti()
+    ti.axcom_push = AsyncMock()
+    context = {"task_instance": ti}
+    if task_state_store is not None:
+        context["task_state_store"] = task_state_store
+    return context
+
+
+def _make_async_mock_agent(mock_hook_cls, **run):
+    """Wire a mock agent whose ``run`` is awaitable into the hook's async path, and return it."""
+    mock_agent = MagicMock(spec=["run", "instrument"])
+    mock_agent.run = AsyncMock(**run)
+    mock_hook_cls.aget_hook = AsyncMock()
+    mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(return_value=mock_agent)
+    return mock_agent
+
+
+@pytest.mark.skipif(not AIRFLOW_V_3_2_PLUS, reason="async tasks need Airflow >= 3.2")
+class TestAgentOperatorAsync:
+    def test_operator_is_synchronous_unless_a_subclass_says_otherwise(self):
+        assert AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm").is_async is False
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_aexecute_awaits_the_run_through_the_async_hook_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        result = await op.aexecute(_make_async_context())
+
+        assert result == "done"
+        mock_hook_cls.aget_hook.assert_awaited_once_with(
+            "my_llm", hook_params={"model_id": None, "fallback_conn_ids": None}
+        )
+        mock_hook_cls.get_hook.assert_not_called()
+        mock_agent.run.assert_awaited_once_with(
+            "run", usage_limits=None, run_id="ti-1", cancellation_token=ANY, usage=ANY
+        )
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.parametrize("async_store", [True, False], ids=["async-store", "store-before-3.4"])
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_successful_run_makes_no_blocking_call_to_the_supervisor(self, mock_hook_cls, async_store):
+        """A real run with a usage budget and a message history: no blocking call reaches the event
+        loop, whether the task state store has async methods (Airflow 3.4+) or not."""
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(_build_priced_response), **kw)
+        )
+        store = _make_async_task_state_store_accessor(async_methods=async_store)
+        context = _make_async_context(task_state_store=store)
+        context["task_instance"].xcom_push.side_effect = AssertionError("blocking xcom_push")
+
+        op = AgentOperator(
+            task_id="test",
+            prompt="run",
+            llm_conn_id="my_llm",
+            usage_limits={"cost_limit": str(PRICED_COST * 2)},
+            message_history=[],
+        )
+        with (
+            patch("airflow.providers.common.ai.operators.agent.AIRFLOW_V_3_4_PLUS", async_store),
+            patch("airflow.providers.common.ai.utils.usage_budget.AIRFLOW_V_3_4_PLUS", async_store),
+        ):
+            result = await op.aexecute(context)
+
+        assert result == "the answer"
+        pushed = [c.kwargs["key"] for c in context["task_instance"].axcom_push.await_args_list]
+        assert pushed == ["run_id", "usage", MODEL_NAME_XCOM_KEY, "message_history"]
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        assert store.ops[-1] == ("delete", USAGE_BUDGET_KEY)
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_failed_run_saves_the_budget_and_reports_its_usage(self, mock_hook_cls):
+        _make_async_mock_agent(mock_hook_cls, side_effect=RuntimeError("boom"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        with pytest.raises(RuntimeError, match="boom"):
+            await op.aexecute(context)
+
+        assert ("set", USAGE_BUDGET_KEY) in store.ops
+        # The failure report reuses the synchronous code, from a worker thread.
+        pushed = {c.kwargs["key"] for c in context["task_instance"].xcom_push.call_args_list}
+        assert pushed == {"run_id", "usage"}
+
+    @pytest.mark.parametrize("feature", ["durable", "enable_hitl_review"])
+    @pytest.mark.asyncio
+    async def test_aexecute_rejects_features_that_block_the_event_loop(self, feature):
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **{feature: True})
+
+        with pytest.raises(ValueError, match="not supported with an async callable"):
+            await op.aexecute(_make_async_context())
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_run_waiting_on_a_tool_approval_pauses_through_the_synchronous_path(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        class Paused(Exception):
+            pass
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(DeferredToolRequests()))
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        with patch.object(
+            AgentOperator,
+            "_pause_for_tool_approval",
+            side_effect=_refused_on_the_event_loop(Mock(side_effect=Paused)),
+        ) as pause:
+            with pytest.raises(Paused):
+                await op.aexecute(context)
+
+        pause.assert_called_once()
+        context["task_instance"].axcom_push.assert_not_awaited()
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_retry_starts_from_the_usage_of_earlier_tries(self, mock_hook_cls, make_mock_run_result):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        context = _make_async_context(task_state_store=store)
+        store.data[USAGE_BUDGET_KEY] = TaskStateStoreUsageBudget(
+            store, max_tries=context["task_instance"].max_tries
+        )._record(RunUsage(requests=3))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", usage_limits={"request_limit": 5}
+        )
+        await op.aexecute(context)
+
+        assert mock_agent.run.await_args.kwargs["usage"].requests == 3
+
+    @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="task state store needs Airflow >= 3.3")
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_approval_transcript_of_an_earlier_try_is_deleted(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("done"))
+        store = _make_async_task_state_store_accessor()
+        store.data[_TOOL_APPROVAL_TRANSCRIPT_KEY] = "[]"
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm")
+        await op.aexecute(_make_async_context(task_state_store=store))
+
+        assert ("delete", _TOOL_APPROVAL_TRANSCRIPT_KEY) in store.ops
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_output_is_returned_as_a_dict(self, mock_hook_cls, make_mock_run_result):
+        class Answer(BaseModel):
+            text: str
+
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result(Answer(text="done")))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=Answer, serialize_output=True
+        )
+
+        assert await op.aexecute(_make_async_context()) == {"text": "done"}
+
+    @requires_typed_xcom
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_a_model_the_worker_does_not_register_is_returned_as_json(
+        self, mock_hook_cls, make_mock_run_result
+    ):
+        """The async path converts the output the way ``execute`` does: a model inside a list output is
+        dumped, since the worker registers no class for an output marker."""
+        _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result([Summary(text="ok")]))
+
+        op = AgentOperator(
+            task_id="test", prompt="run", llm_conn_id="my_llm", output_type=ToolOutput(list[Summary])
+        )
+
+        assert await op.aexecute(_make_async_context()) == [{"text": "ok", "score": 0.0}]
+
+    @pytest.mark.parametrize(
+        ("kwargs", "emits"),
+        [
+            pytest.param({"message_history": []}, True, id="seeded"),
+            pytest.param({"emit_message_history": True}, True, id="opt-in"),
+            pytest.param({"message_history": [], "emit_message_history": False}, False, id="opt-out"),
+        ],
+    )
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_the_transcript_push_follows_emit_message_history(
+        self, mock_hook_cls, kwargs, emits, make_mock_run_result
+    ):
+        mock_agent = _make_async_mock_agent(mock_hook_cls, return_value=make_mock_run_result("ok"))
+        mock_agent.run.return_value.all_messages.return_value = _sample_history()
+        context = _make_async_context()
+
+        op = AgentOperator(task_id="test", prompt="run", llm_conn_id="my_llm", **kwargs)
+        await op.aexecute(context)
+
+        pushes = {
+            c.kwargs["key"]: c.kwargs["value"] for c in context["task_instance"].axcom_push.await_args_list
+        }
+        assert ("message_history" in pushes) is emits
+        if emits:
+            assert len(ModelMessagesTypeAdapter.validate_json(pushes["message_history"])) == 2
+
+    @pytest.mark.asyncio
+    @patch("airflow.providers.common.ai.operators.agent.PydanticAIHook", autospec=True)
+    async def test_runs_overlap_on_one_event_loop(self, mock_hook_cls):
+        async def slow(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
+            await asyncio.sleep(0.2)
+            return ModelResponse(parts=[TextPart(content="the answer")])
+
+        mock_hook_cls.aget_hook = AsyncMock()
+        mock_hook_cls.aget_hook.return_value.acreate_agent = AsyncMock(
+            side_effect=lambda **kw: Agent(FunctionModel(slow), **kw)
+        )
+        ops = [AgentOperator(task_id=f"test_{i}", prompt="run", llm_conn_id="my_llm") for i in range(8)]
+
+        loop = asyncio.get_running_loop()
+        started = loop.time()
+        results = await asyncio.gather(*(op.aexecute(_make_async_context()) for op in ops))
+
+        assert results == ["the answer"] * 8
+        # Eight runs of 0.2 s each take 1.6 s one after the other.
+        assert loop.time() - started < 1.0

Review Comment:
   Added in b7657566a7: `slow` records its thread and the test asserts all eight runs ran on the loop's own thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #74201:
URL: https://github.com/apache/airflow/pull/74201#issuecomment-6475923115

   <!-- spot-interruption-notice run=40731779446 -->
   ⚠️ 1 job(s) of [attempt 1](https://github.com/apache/airflow/actions/runs/40731779446/attempts/1) were stopped because AWS reclaimed their EC2 Spot instances. These are not test failures.
   
   - [`Provider distributions tests / Compat 3.3.2:P3.11:amazon...standard`](https://github.com/apache/airflow/actions/runs/40731779446/job/122974715374)
   
   RunsOn re-runs them automatically when the run finishes; no action is needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "kaxil (via GitHub) via commits" <co...@airflow.apache.org>.
kaxil commented on code in PR #74201:
URL: https://github.com/apache/airflow/pull/74201#discussion_r4483787675


##########
providers/common/ai/src/airflow/providers/common/ai/decorators/agent.py:
##########
@@ -106,6 +117,19 @@ def execute(self, context: Context) -> Any:
         self.prompt = self.render_template(self.prompt, context)
         return AgentOperator.execute(self, context)
 
+    async def aexecute(self, context: Context) -> Any:
+        context_merge(context, self.op_kwargs)
+        kwargs = determine_kwargs(self.python_callable, self.op_args, context)
+
+        self.prompt = await self.python_callable(*self.op_args, **kwargs)
+
+        validate_prompt(self.prompt, decorator_name="@task.agent")
+
+        # Rendering a template can read a Variable or a Connection with a blocking call to
+        # the supervisor, which must not happen on the event loop.
+        await asyncio.to_thread(self.render_template_fields, context)

Review Comment:
   And we actually just fixed it recenrtly on main



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add native async execution to AgentOperator and @task.agent [airflow]

Posted by "kaxil (via GitHub) via commits" <co...@airflow.apache.org>.
kaxil commented on PR #74201:
URL: https://github.com/apache/airflow/pull/74201#issuecomment-6836599881

   Should've asked this in the first round. Since `@task.agent` is one run per task, the many-runs-on-one-loop numbers don't apply to the decorator, and an `async def` gives up `durable` and HITL. Trying to figure out if async has any benefits here at all. What's the use case you have in mind?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org