You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2026/10/14 14:02:59 UTC

(airflow) 01/01: Document progressive tool disclosure for Common AI toolsets

This is an automated email from the ASF dual-hosted git repository.

kaxil pushed a commit to branch commonai-progressive-tool-disclosure
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit b38c98ecb1e7daef8abdf29227e04bdf87335c1e
Author: Kaxil Naik <ka...@apache.org>
AuthorDate: Wed Oct 14 13:52:33 2026 +0100

    Document progressive tool disclosure for Common AI toolsets
---
 providers/common/ai/docs/features.rst              |   4 +
 providers/common/ai/docs/operators/agent.rst       |   7 +-
 providers/common/ai/docs/stability.rst             |   5 +-
 providers/common/ai/docs/toolsets/index.rst        |   6 +-
 providers/common/ai/docs/toolsets/logging.rst      |   4 +-
 .../ai/docs/toolsets/progressive_disclosure.rst    | 178 +++++++++++++++++++++
 providers/common/ai/docs/toolsets/skills.rst       |  10 +-
 providers/common/ai/pyproject.toml                 |   5 +-
 .../example_progressive_tool_disclosure.py         |  98 ++++++++++++
 9 files changed, 302 insertions(+), 15 deletions(-)

diff --git a/providers/common/ai/docs/features.rst b/providers/common/ai/docs/features.rst
index 2526b79ae3c..737d9c8c2ef 100644
--- a/providers/common/ai/docs/features.rst
+++ b/providers/common/ai/docs/features.rst
@@ -135,6 +135,10 @@ checklist.
    * - :doc:`ManagedAgentToolset <toolsets/managed_agent>`
      - Sends a prompt to an agent on Bedrock AgentCore or Vertex AI Agent Engine.
      - The Amazon or Google provider's ``common.ai`` extra
+   * - :doc:`Progressive tool disclosure <toolsets/progressive_disclosure>`
+     - Hides a toolset's tools from the model until it searches for them, with
+       ``.defer_loading()`` on any toolset.
+     - None
 
 Control output and spend
 ------------------------
diff --git a/providers/common/ai/docs/operators/agent.rst b/providers/common/ai/docs/operators/agent.rst
index 7e9b41d463f..fbb7b542989 100644
--- a/providers/common/ai/docs/operators/agent.rst
+++ b/providers/common/ai/docs/operators/agent.rst
@@ -378,9 +378,10 @@ Parameters
 - ``enable_tool_logging``: Wrap the assembled function toolset in
   :class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` so that
   tools supplied through ``toolsets=``, ``agent_params={"tools": [...]}``, and
-  capabilities are logged in real time. Output tools, provider-native tools, and
-  tools another capability adds through its own wrapper (code mode's ``run_code``,
-  ToolSearch's ``search_tools``) are not covered. Default ``True``.
+  capabilities are logged in real time, including tools the model found through
+  :ref:`tool search <howto/toolset:progressive-disclosure>`. Output tools, provider-native
+  tools, and tools another capability adds through its own wrapper (code mode's
+  ``run_code``, ToolSearch's ``search_tools``) are not covered. Default ``True``.
 - ``agent_params``: Additional keyword arguments passed to the pydantic-ai
   ``Agent`` constructor (e.g. ``retries``, ``model_settings``).
 - .. _agent-usage-budget:
diff --git a/providers/common/ai/docs/stability.rst b/providers/common/ai/docs/stability.rst
index 5b2e2a61f63..c9733255cd5 100644
--- a/providers/common/ai/docs/stability.rst
+++ b/providers/common/ai/docs/stability.rst
@@ -223,8 +223,9 @@ Everything this provider ships that is not in the table above is experimental.
      - Thin integrations of ``pydantic-ai-harness`` and ``pydantic-ai-shields``, which are
        before their 1.0 releases.
    * - The Agent Skills toolset (:doc:`toolsets/skills`)
-     - It may move from ``pydantic-ai-skills`` to the on-demand loading Pydantic AI now has
-       itself (`pydantic/pydantic-ai#5230 <https://github.com/pydantic/pydantic-ai/pull/5230>`__).
+     - It is built on ``pydantic-ai-skills``, a community package. Whether it moves to
+       ``pydantic-ai-harness``'s ``Skills`` once that loads skill resources and scripts is
+       still open.
    * - :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset` and its
        backends (:doc:`sandbox/index`)
      - ``durable=True`` cannot be used with it yet, the ``sbx`` backend leaves its sandbox
diff --git a/providers/common/ai/docs/toolsets/index.rst b/providers/common/ai/docs/toolsets/index.rst
index c69bfffddfb..faa2085fa6d 100644
--- a/providers/common/ai/docs/toolsets/index.rst
+++ b/providers/common/ai/docs/toolsets/index.rst
@@ -67,13 +67,14 @@ with is resolved in the worker and is not part of any of that.
      - ``timeout`` for each call; what the remote agent may touch is set at the
        vendor
 
-Two controls work on every toolset. ``.approval_required()`` pauses the task before a
+Three controls work on every toolset. ``.approval_required()`` pauses the task before a
 matching call runs, until a person approves or rejects it on the **Required Actions**
 page. It needs Airflow 3.3 or later, pauses at every step that calls a matching tool on
 Airflow 3.4 and later (once per Dag run on 3.3), and does not combine with ``durable=True`` or a ``SandboxToolset`` that provisions its
 own sandbox (:doc:`../tool_approval` lists every limit). ``.filtered()`` drops tools
 from what the model is offered, as the :ref:`MCP guide <howto/toolset:mcp-filtered>`
-shows.
+shows. ``.defer_loading()`` hides a toolset's tools from the model until it searches for
+them, which pays off on a toolset with many tools (see :doc:`progressive_disclosure`).
 
 :doc:`logging` and :doc:`langchain` do not reach a system of their own: one wraps a
 toolset to log its calls, the other converts a toolset for a LangChain agent.
@@ -92,6 +93,7 @@ toolset to log its calls, the other converts a toolset for a LangChain agent.
     Vendor-managed agents <managed_agent>
     LangChain tools <langchain>
     Tool call logging <logging>
+    Progressive tool disclosure <progressive_disclosure>
 
 Importing toolsets
 ------------------
diff --git a/providers/common/ai/docs/toolsets/logging.rst b/providers/common/ai/docs/toolsets/logging.rst
index 6cba50b98b5..a83d3b0984f 100644
--- a/providers/common/ai/docs/toolsets/logging.rst
+++ b/providers/common/ai/docs/toolsets/logging.rst
@@ -35,7 +35,9 @@ toolsets, nested capabilities, and MCP toolsets. Output tools such as
 ``final_result`` and provider-native tools, including native MCP, are not
 covered by Airflow's real-time tool-call logging. Tools added by another
 capability through its own wrapper toolset, such as ToolSearch's
-``search_tools`` and CodeMode's ``run_code``, are also not covered.
+``search_tools`` and CodeMode's ``run_code``, are also not covered. A tool hidden with
+``.defer_loading()`` is logged once the model finds it, and its result is masked like
+any other (see :doc:`progressive_disclosure`).
 
 ``AgentOperator`` adds ``ToolLoggingCapability`` automatically when
 ``enable_tool_logging=True``. If ``capabilities=`` already contains one,
diff --git a/providers/common/ai/docs/toolsets/progressive_disclosure.rst b/providers/common/ai/docs/toolsets/progressive_disclosure.rst
new file mode 100644
index 00000000000..24b1a62fcff
--- /dev/null
+++ b/providers/common/ai/docs/toolsets/progressive_disclosure.rst
@@ -0,0 +1,178 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _howto/toolset:progressive-disclosure:
+
+Progressive tool disclosure
+===========================
+
+Every tool an agent can call costs input tokens on every model request: its name,
+description and argument schema are sent each time, whether the model uses the tool or
+not. ``.defer_loading()`` hides a toolset's tools until the model searches for them, so
+an agent can carry a wide toolset and pay mostly for the tools a run looks for. It is a
+Pydantic AI method that every toolset has, including the ones this provider ships. It
+pays off on a toolset with many tools, and does not on one with a handful, such as
+``SQLToolset`` (see :ref:`progressive-disclosure-when-not`). A tool no run needs is better
+left out of the toolset altogether, with ``allowed_methods`` or ``.filtered()``; defer the
+ones only some runs need.
+
+Hide a toolset's tools until the model needs them
+-------------------------------------------------
+
+This agent can call nine read-only ``S3Hook`` methods on one bucket, and the model sees
+none of them until it searches:
+
+.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
+    :language: python
+    :dedent: 4
+    :start-after: [START howto_toolset_defer_loading]
+    :end-before: [END howto_toolset_defer_loading]
+
+How the model finds a hidden tool depends on the model:
+
+- On models with native tool search, the deferred tools are sent to the provider with
+  ``defer_loading`` set, and the provider keeps their definitions out of the model's
+  context and searches them itself. Pydantic AI uses native search for Claude Sonnet 4.5,
+  Opus 4.5 and Haiku 4.5 and the later Claude families it lists, and for GPT-5.4 and later
+  through the OpenAI Responses API; its
+  `tool search guide <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/#tool-search>`__
+  has the details.
+- On every other model, the hidden tools are left out of the request. The model is
+  offered one ``search_tools`` tool that matches keywords against tool names and
+  descriptions, and the tools it finds are sent with the next request.
+
+Pydantic AI picks the path from a list of model name prefixes in the installed version, so
+a model family that version does not list gets the local search. For example, Pydantic AI
+2.48 lists ``claude-haiku-4-5`` but no Haiku 5 model, so ``claude-haiku-5-5`` takes the
+local path.
+
+Either way, a tool the model has found stays available while the search that found it
+is in the run's message history.
+
+Keep the common tools visible
+-----------------------------
+
+A search costs a model request. When most runs use the same few tools, pass the names of
+the tools to hide, and leave the rest in the request from the start:
+
+.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
+    :language: python
+    :dedent: 4
+    :start-after: [START howto_toolset_defer_loading_some]
+    :end-before: [END howto_toolset_defer_loading_some]
+
+The names are the tool names the model sees, so they include ``tool_prefix``. A name that
+matches no tool is ignored without an error, so a misspelled name leaves that tool visible.
+
+What it saves
+-------------
+
+These counts are the tokens in the tool definitions of the first model request, counted
+with ``tiktoken``'s ``o200k_base`` encoding on a model without native tool search. Each
+model provider counts tokens its own way, so its numbers differ somewhat.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 52 16 16 16
+
+   * - Toolset
+     - All tools sent
+     - ``.defer_loading()``
+     - ``s3_list_keys`` and ``s3_read_key`` visible
+   * - The nine ``S3Hook`` methods above
+     - 1,059
+     - 163
+     - 694
+   * - 25 ``S3Hook`` methods, writes included
+     - 4,609
+     - 163
+     - 761
+
+The 163 tokens are ``search_tools`` itself, so the saving grows with the number of tools
+hidden. On a model with native tool search there is no ``search_tools``, and the
+provider keeps the deferred tools out of the model's context until they are found.
+
+What a run looks like
+---------------------
+
+The first example, run on ``claude-haiku-5-5`` (which took the local search), searched
+once, then called ``s3_list_keys`` once and ``s3_read_key`` twice. This is the
+``AgentOperator`` part of its task log, with timestamps and logger names trimmed:
+
+.. code-block:: text
+
+    ::group::Tool call: s3_list_keys
+    Tool s3_list_keys returned in 0.20s
+    ::endgroup::
+    ::group::Tool call: s3_read_key
+    Tool s3_read_key returned in 0.08s
+    ::endgroup::
+    ::group::Tool call: s3_read_key
+    Tool s3_read_key returned in 0.01s
+    ::endgroup::
+    ::group::LLM run complete: model=claude-haiku-5-5, requests=4, tool_calls=4, input_tokens=6683, output_tokens=474, total_tokens=7157
+    LLM prompt cache: cache_read_tokens=3894, cache_write_tokens=2779
+    Tool call sequence: search_tools -> s3_list_keys -> s3_read_key -> s3_read_key
+    ::endgroup::
+
+``search_tools`` has no ``Tool call`` group of its own, because tool call logging does not
+cover tools another capability adds. It shows only in the sequence at the end of the run,
+and it is one of the four tool calls counted.
+
+The second example, with ``s3_list_keys`` and ``s3_read_key`` visible from the start, made
+the same three S3 calls without a search, in one model request fewer:
+
+.. code-block:: text
+
+    ::group::LLM run complete: model=claude-haiku-5-5, requests=3, tool_calls=3, input_tokens=4633, output_tokens=394, total_tokens=5027
+    LLM prompt cache: cache_read_tokens=2880, cache_write_tokens=1745
+    Tool call sequence: s3_list_keys -> s3_read_key -> s3_read_key
+    ::endgroup::
+
+The local search matches each keyword on its own, so one search can reveal more tools than
+the model goes on to use. In a run where the model searched for ``list keys read key``,
+the search revealed six of the nine S3 tools, and every request after it carried all six.
+
+What stays the same
+-------------------
+
+A tool the model found is the same tool, called the same way:
+
+- Its result passes through Airflow's secret masker before the model sees it, as every
+  tool result does (see :doc:`../agent_security`).
+- Its calls are logged like any other tool's (see :doc:`logging`).
+- With ``durable=True``, a retry replays the model requests and tool calls that came
+  after a search; only the step that failed runs again (see :doc:`../durable_execution`).
+- A templated connection ID on the toolset is rendered for each task instance, as it is
+  without ``.defer_loading()``.
+
+.. _progressive-disclosure-when-not:
+
+When it does not help
+---------------------
+
+``SQLToolset`` has four tools whatever the database holds: 488 tokens of tool
+definitions, against 163 for ``search_tools`` in their place. Hiding them saves 325
+tokens a request and costs a search request in nearly every run, since the model needs
+the tools to answer anything. On a warehouse with hundreds of tables, ``list_tables``
+output grows with the table count; limit it with ``allowed_tables`` (see
+:ref:`sql-toolset-restricted`).
+
+``AgentSkillsToolset`` has four tools, and its list of skills goes into the agent's
+instructions, which ``.defer_loading()`` does not touch. For 30 skills that list was
+2,129 tokens against 907 for the tools. The toolset already loads each skill's body only
+when the model asks for it.
diff --git a/providers/common/ai/docs/toolsets/skills.rst b/providers/common/ai/docs/toolsets/skills.rst
index 52398c8a064..ea7b33c3245 100644
--- a/providers/common/ai/docs/toolsets/skills.rst
+++ b/providers/common/ai/docs/toolsets/skills.rst
@@ -33,11 +33,11 @@ prompt until the model decides it needs one, so a large skill library costs few
 tokens until used (progressive disclosure).
 
 It is backed by the community `pydantic-ai-skills
-<https://github.com/DougTrajano/pydantic-ai-skills>`__ package (MIT). Pydantic AI now
-has on-demand loading of its own (`pydantic/pydantic-ai#5230
-<https://github.com/pydantic/pydantic-ai/pull/5230>`__), and this toolset may move to it.
-Install the optional
-extra to use it:
+<https://github.com/DougTrajano/pydantic-ai-skills>`__ package (MIT). ``pydantic-ai-harness``'s
+`Skills <https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/skills>`__
+capability also loads Agent Skills, as on-demand capabilities, but only their ``SKILL.md``
+instructions: no resources and no ``run_skill_script``.
+Install the optional extra to use this toolset:
 
 .. code-block:: bash
 
diff --git a/providers/common/ai/pyproject.toml b/providers/common/ai/pyproject.toml
index 9aebf776494..71a3e1ac8e3 100644
--- a/providers/common/ai/pyproject.toml
+++ b/providers/common/ai/pyproject.toml
@@ -132,8 +132,9 @@ dependencies = [
 "shields" = ["pydantic-ai-shields>=0.3.4"]
 # Agent Skills (agentskills.io) support. pydantic-ai-skills provides the toolset;
 # the git provider supplies GitHook + GitPython for cloning GitSkills with
-# credentials from a `git` connection. Native progressive disclosure is tracked
-# upstream in pydantic/pydantic-ai#5230; revisit this extra once that lands.
+# credentials from a `git` connection. pydantic-ai-harness's Skills capability is not
+# a replacement: it loads only SKILL.md instructions, with no skill resources and no
+# run_skill_script.
 "skills" = [
     "apache-airflow-providers-git>=0.4.0",
     "pydantic-ai-skills>=1.2.0",
diff --git a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
new file mode 100644
index 00000000000..12085b470a5
--- /dev/null
+++ b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example Dags: an agent whose S3 tools stay hidden until the model searches for them."""
+
+from __future__ import annotations
+
+from airflow.providers.common.ai.operators.agent import AgentOperator
+from airflow.providers.common.ai.toolsets import HookToolset
+from airflow.providers.common.compat.sdk import dag
+
+try:
+    from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+except ImportError:
+    S3Hook = None  # type: ignore[assignment,misc]
+
+
+if S3Hook is not None:
+    # [START howto_toolset_defer_loading]
+    # Nine read-only S3Hook methods, one tool each.
+    REPORT_METHODS = [
+        "list_keys",
+        "list_prefixes",
+        "check_for_key",
+        "check_for_prefix",
+        "check_for_wildcard_key",
+        "check_for_bucket",
+        "head_object",
+        "read_key",
+        "get_bucket_tagging",
+    ]
+
+    @dag(tags=["example"])
+    def example_defer_loading():
+        AgentOperator(
+            task_id="answer_from_reports",
+            prompt="Summarize the reports under 2026/09/.",
+            llm_conn_id="pydanticai_default",
+            toolsets=[
+                # Hide every tool until the model searches for it.
+                HookToolset(
+                    S3Hook(aws_conn_id="aws_reports_reader"),
+                    allowed_methods=REPORT_METHODS,
+                    pinned_arguments={"bucket_name": "acme-reports"},
+                    tool_prefix="s3",
+                ).defer_loading()
+            ],
+        )
+
+    # [END howto_toolset_defer_loading]
+
+    example_defer_loading()
+
+    # [START howto_toolset_defer_loading_some]
+    @dag(tags=["example"])
+    def example_defer_loading_some():
+        AgentOperator(
+            task_id="answer_from_reports",
+            prompt="Summarize the reports under 2026/09/.",
+            llm_conn_id="pydanticai_default",
+            toolsets=[
+                HookToolset(
+                    S3Hook(aws_conn_id="aws_reports_reader"),
+                    # The nine methods of the previous example.
+                    allowed_methods=REPORT_METHODS,
+                    pinned_arguments={"bucket_name": "acme-reports"},
+                    tool_prefix="s3",
+                ).defer_loading(
+                    # Hide these seven; s3_list_keys and s3_read_key, which most runs use, stay visible.
+                    [
+                        "s3_list_prefixes",
+                        "s3_check_for_key",
+                        "s3_check_for_prefix",
+                        "s3_check_for_wildcard_key",
+                        "s3_check_for_bucket",
+                        "s3_head_object",
+                        "s3_get_bucket_tagging",
+                    ]
+                )
+            ],
+        )
+
+    # [END howto_toolset_defer_loading_some]
+
+    example_defer_loading_some()