You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zozo123 (via GitHub)" <gi...@apache.org> on 2026/08/16 12:11:33 UTC
[PR] Add OpenSandbox backend for sandbox tools [airflow]
zozo123 opened a new pull request, #71676:
URL: https://github.com/apache/airflow/pull/71676
Adds a self-hosted OpenSandbox backend for `SandboxToolset`, so Airflow workers can use a remote Docker or Kubernetes sandbox service without access to the container runtime.
### What changed
- maps `SandboxSpec` environment and network policy at creation, preserving the fail-closed contract;
- streams byte-bounded command output and uses OpenSandbox's native file APIs;
- applies server-side command deadlines and a server-side sandbox lifetime;
- resolves the optional SDK and a generic Airflow connection lazily; and
- documents the `sandbox-opensandbox` extra and adds unit plus system coverage.
### Validation
- 38 focused unit tests pass with the minimum/current `opensandbox==0.1.15`;
- a real OpenSandbox Docker server test passed with authenticated creation, deny-all egress, all backend operations, output truncation, timeout, and idempotent teardown;
- the Common AI provider wheel builds with the new extra and backend; and
- Ruff, formatting, codespell, YAML/Markdown, provider metadata, dependency, documentation, and system-test hooks pass locally.
Related: #68847
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Codex (GPT-5.6)
Generated-by: Codex (GPT-5.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on code in PR #71676:
URL: https://github.com/apache/airflow/pull/71676#discussion_r4003838881
##########
providers/common/ai/docs/toolsets.rst:
##########
@@ -793,6 +793,56 @@ Constructor parameters:
guarantee this backend cannot make. Set ``"deny-all"`` after running
``sbx policy init deny-all``, or ``"allow-all"`` to state that egress is open.
+OpenSandbox backend
Review Comment:
Done — recording is in the PR thread: a full end-to-end run against a real OpenSandbox server (`opensandbox/server:v0.2.3`, SDK `0.1.16`, Docker runtime, arm64).
It covers API-key enforcement, all four `SandboxToolset` tools driven by a real pydantic-ai agent, this PR's own system-test Dag task, the `SandboxSpec` fail-closed egress matrix, and teardown.
Running it surfaced two genuine defects, both now fixed with test coverage: the system test could never construct (30s ceiling vs the 60s default), and command output was losing every line break. Details in the PR comment.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5774742637
I went back through each round-2 thread against the `opensandbox` 1.1.0 source rather than the fixtures, and the fixes hold: the dispatcher records `error` and `complete` regardless of `skip_accumulation`, so the exit-code path sees real data; a plain 404 on `connect` is not retried by the readiness budget, so `destroy` on a gone sandbox returns at once; `get_egress_policy`, the `metadata` kwarg and `skip_health_check` all exist with the shapes used here. Nothing blocking from me. Three non-blocking notes, here rather than as a review round:
1. `docs/sandbox/backends.rst` L228, the "Command timeouts" bullet says a timeout destroys an OpenSandbox sandbox and its files. Your own measurement above (`sleep 60` on a 5s budget gives `sandbox_terminated=False`) shows a server-enforced timeout leaves the sandbox and its files intact, and only a stalled stream destroys it. OpenSandbox belongs on the Modal side of that sentence, with the stall case as the exception.
2. `_get_network_policy` sends `NetworkPolicy(defaultAction="allow")` for `block_network=False`. The server refuses any create that carries a `networkPolicy` when `egress.image` is not configured (`ensure_egress_configured` in `server/opensandbox_server/services/validators.py` returns early only on a falsy policy), and the Docker runtime refuses one under `network_mode=host` or a user-defined network as well (`_ensure_network_policy_support` in `services/docker/networking.py`). So a deployment without the sidecar cannot run an open-network spec even though there is nothing to enforce. Returning `None` for the open case would let those deployments work, and `create` already skips the read-back for it.
3. The Modal backend's extra is `modal`; this one is `sandbox-opensandbox`. Extras are permanent public surface, so worth deciding now whether it should be `opensandbox` to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5661805017
Yes — ran it end to end against a real OpenSandbox server (Docker runtime, `opensandbox/server:v0.2.3`, SDK `0.1.16`, arm64). Recording below: every sandbox in it is a real container created and destroyed through the server's API, and the egress results come from a stdlib socket probe *inside* the sandbox, so a failure can't be mistaken for a missing tool.
The clip covers: API-key enforcement (401 → 200), all four `SandboxToolset` tools driven by a real pydantic-ai agent, this PR's own system-test Dag task, the `SandboxSpec` egress matrix, and teardown.
Trying it out turned up two things this PR had wrong. Both are fixed in the push above:
**The system test could never have run.** It asked for a 30s command ceiling while leaving `default_command_timeout` at 60s, which `SandboxToolset` rejects at construction — it failed before reaching a sandbox. `example_sandbox_toolset_sbx.py` on `main` has the same bug; I left it alone to keep this PR scoped, happy to send it separately.
**Command output lost every line break.** execd streams one message per output line with the delimiter stripped, and the backend concatenated them, so `printf 'A\nB\nC\n'` came back as `'ABC'`. Everything reached the model as a single run-on line — and since command output is truncated by whole lines, `max_output_lines` was inert for this backend. Fixed with a regression test, verified byte-for-byte against the live server including the blank-line case.
Also rebased onto `main` with `uv.lock` regenerated, and moved to `opensandbox>=0.1.16`. 39 unit tests pass on both 0.1.15 and 0.1.16.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5781541307
All three notes from your pass earlier today are addressed; the extra name is settled.
**1. Command-timeout docs.** `docs/sandbox/backends.rst` now puts OpenSandbox on the Modal side of that sentence, with the stall case as the exception:
> A timeout destroys an `sbx` sandbox and its files; Modal and a server-enforced OpenSandbox timeout preserve the sandbox and files. OpenSandbox destroys it only if the command event stream itself stalls past the client-side grace period.
**2. Open-network specs.** `_get_network_policy` returns `None` when `block_network=False` and no allowlist is set, so a deployment without the egress sidecar can run an open-network spec. `allow_egress_to_cidrs` is now refused up front, naming the dns+nft read-back gap as the reason rather than accepting a policy it cannot prove is enforced.
**3. Extra name.** Renamed to `opensandbox`, matching `modal`. Applied across `providers/common/ai/pyproject.toml`, `docs/installation.rst`, `docs/index.rst`, `README.rst` and the SDK-missing error message; no occurrence of the old spelling remains.
While re-checking that rename I found it had shortened the extras-table cell by eight characters without re-padding the row, so the second column started inside the first column's span. docutils rejects that with "Malformed table. Text in column margin" and parses zero rows, which drops the whole Optional dependencies table from the rendered docs and from `README.rst` rather than just misaligning one line. Fixed in d12cf0e.
64/64 unit tests pass.
Drafted-by: Claude Code (Claude Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #71676:
URL: https://github.com/apache/airflow/pull/71676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5774112201
Pushed the round for the 2026-09-17 review; replies are inline on each thread.
Behaviour changes, all confined to the OpenSandbox backend: the network policy is read back after create and a sandbox the server did not restrict is destroyed; a stalled command stream is abandoned 30s past its budget with the sandbox destroyed and `sandbox_terminated` set; an exit status the SDK could not parse is no longer a terminal error; oversized reads report the file's real size; listings are capped with guidance; a connection without a host is refused instead of falling back to `localhost:8080`; reconnect time is not charged to the command; `destroy` skips the readiness poll; and every sandbox carries `created-by: airflow` metadata.
Two threads asked for evidence against a real server rather than fixtures. Both are now covered, against a local `opensandbox-server` on Docker:
| case | result |
| --- | --- |
| server with **no egress sidecar**, default deny-all spec | server refuses the create -- `400 [SANDBOX::INVALID_PARAMETER] egress.image must be configured when networkPolicy is provided` -- surfaced as `SandboxTerminalError`. Fails closed at create; the read-back stays as the backstop for a build that would accept it silently. |
| real timeout, 5s budget against `sleep 60` | `elapsed=6.0s exit_code=-1 timed_out=True sandbox_terminated=False`, stderr `signal: killed` |
| signal kill, `sh -c 'kill -9 $$'` on a 30s budget | `elapsed=1.0s exit_code=-1 timed_out=False sandbox_terminated=False`, stderr `signal: killed` |
The third is the one the `exit_code` thread was about: execd words `error.value` as the prose `signal: killed`, which `_infer_foreground_exit_code` cannot parse. It is now an ordinary failed command, and because it died well inside its budget it is not misreported as a timeout either.
The system test also passed end to end against that server (`1 passed in 20.83s`), exercising create, two commands, write, read, mkdir, list, the egress policy read-back and teardown, with no containers left behind.
Rebased onto current `main`, which reorganised the provider docs twice today, so the backend is documented in `docs/sandbox/backends.rst` beside Modal and `sbx` rather than in `toolsets.rst`. Also in this push: the SDK moves to `opensandbox>=1.1.0`, which I checked is API- and behaviour-identical for everything this backend touches, including the disabled SSE read timeout, `_infer_foreground_exit_code` and the unbounded frame buffer; it needs a temporary `exclude-newer-package` override until the rolling window passes 2026-09-21. Two packaging fixes came out of that review: the SDK had been listed under the `llamaindex` extra rather than `[dependency-groups].dev`, which both dragged it into `[llamaindex]` and made the unit tests `importorskip` away in the main lane, and the extras table in `docs/index.rst` had mismatched column rules with no `sandbox-opensandbox` row.
63 unit tests, ruff and the applicable prek hooks pass locally; mypy is clean on the changed files with the SDK installed.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5790779018
CI is green on 425079a1 — 295 passed, 0 failed — and the extra name you asked me to settle before
merge is now `opensandbox`, matching `modal`.
The five threads still open are the round-2 ones; each has a reply and the fixes are in the current
head: `_CREATED_BY_METADATA` applied on every create (L332), an empty connection host refused
(L243), `_get_sandbox` moved ahead of the elapsed-time clock (L363/L366),
`_LIST_DIRECTORY_MAX_ENTRIES = 10_000` (L58/L507), and `destroy` reconnecting with
`skip_health_check=True` (L528). Happy to resolve them myself if you would rather — I left them for
you since they are your comments.
Anything else you want before this goes in?
Drafted-by: Claude Code (Claude Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5672080895
> Yes — ran it end to end against a real OpenSandbox server (Docker runtime, `opensandbox/server:v0.2.3`, SDK `0.1.16`, arm64).
>
> <img alt="opensandbox-airflow" height="924" src="https://private-user-images.githubusercontent.com/553010/651222971-a51f16e2-62e9-4561-a08f-35f5734f257a.gif?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODk0Mjc4OTcsIm5iZiI6MTc4OTQyNzU5NywicGF0aCI6Ii81NTMwMTAvNjUxMjIyOTcxLWE1MWYxNmUyLTYyZTktNDU2MS1hMDhmLTM1ZjU3MzRmMjU3YS5naWY_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwOTE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDkxNFQyMzEzMTdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wNjZhOTgyNDhlNTMyZmY3ZjEwMDRhZmU3YTgyZmM3ZjRlMTc4YjQ0MTVhNTY4Zjk1YzM5ZjhiNzMwZWUwZDExJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZyZXNwb25zZS1jb250ZW50LXR5cGU9aW1hZ2UlMkZnaWYifQ.GIAjm33P9rya3GncKQeatEBy8E_zmAmjkHQZJabW1PU">
> Every sandbox in the clip is a real container created and destroyed through the server's API, and the egress results come from a stdlib socket probe _inside_ the sandbox, so a failure can't be mistaken for a missing tool. It covers API-key enforcement (401 → 200), all four `SandboxToolset` tools driven by a real pydantic-ai agent, this PR's own system-test Dag task, the `SandboxSpec` egress matrix, and teardown.
>
> Trying it out turned up two things this PR had wrong. Both are fixed in the push above:
>
> **The system test could never have run.** It asked for a 30s command ceiling while leaving `default_command_timeout` at 60s, which `SandboxToolset` rejects at construction — it failed before reaching a sandbox. `example_sandbox_toolset_sbx.py` on `main` has the same bug; I left it alone to keep this PR scoped, happy to send it separately.
>
> **Command output lost every line break.** execd streams one message per output line with the delimiter stripped, and the backend concatenated them, so `printf 'A\nB\nC\n'` came back as `'ABC'`. Everything reached the model as a single run-on line — and since command output is truncated by whole lines, `max_output_lines` was inert for this backend. Fixed with a regression test, verified byte-for-byte against the live server including the blank-line case.
>
> Also rebased onto `main` with `uv.lock` regenerated, and moved to `opensandbox>=0.1.16`. 39 unit tests pass on both 0.1.15 and 0.1.16.
>
> Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
Could you post what the task logs in Airflow UI looks like too please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on code in PR #71676:
URL: https://github.com/apache/airflow/pull/71676#discussion_r4070142641
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
Review Comment:
Agreed, and it is ours now. After a create whose policy is deny-by-default the backend reads the enforced policy back with `get_egress_policy()` and requires `default_action == "deny"` plus exactly the requested allow targets; a mismatch, or a read-back that fails because there is no sidecar endpoint, destroys the sandbox and raises `SandboxTerminalError`. An open-network spec skips the read-back, since there is nothing to enforce. The class docstring and the docs now state the guarantee as the backend's rather than the server's.
You asked whether I had tried a server without the sidecar. I have now, against a local `opensandbox-server` with the `[egress]` section removed entirely. It is better than either of us assumed: the server refuses the create itself rather than quietly provisioning open egress.
```
create(SandboxSpec()) # block_network=True, the default
-> 400 [SANDBOX::INVALID_PARAMETER] egress.image must be configured when networkPolicy is provided
-> SandboxTerminalError: OpenSandbox could not create a sandbox (HTTP 400).
```
So on this server version the deny-all path fails closed at create and never reaches the read-back. The read-back stays as the backstop for a build that would accept the policy and not enforce it, which is the case neither of us can rule out from the SDK.
Tests: `test_enforced_deny_policy_is_read_back_before_the_sandbox_is_handed_out`, and `test_unenforced_policy_destroys_the_sandbox_and_is_terminal` over open egress, a wider allowlist, a missing rule and a 404 from the sidecar endpoint.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
Review Comment:
Done. Every create passes `metadata={"created-by": "airflow", "name": "airflow-sandbox-<hex>"}`, the name from `_new_sandbox_name()` so it matches the `sbx` correlation prefix, and `SandboxFilter(metadata=...)` can select on it. The docs say so. Test: `test_sandboxes_are_tagged_as_airflows`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
Review Comment:
Done. `commands.run` runs on a daemon thread and the backend waits `timeout + _EXEC_GRACE` (30s, the same constant `sbx` uses). Past that it evicts the handle, destroys the sandbox -- which is what ends the stalled stream -- logs if the destroy is refused, and returns `exit_code=-1, timed_out=True, sandbox_terminated=True` with whatever the tails had collected, so the toolset's fresh-sandbox recovery works here too.
Worth separating the two paths, since I measured both today. A server-enforced timeout is not a stall: with a 5s budget against `sleep 60` the server returns on its own and the sandbox survives, so the result is `timed_out=True, sandbox_terminated=False` after 6.0s. The abandon-and-destroy path above is only for a stream that stops producing events.
You are right about `max_output_bytes` too, and it is not fixable here: `_BoundedTail` holds to the cap for what it is handed, but the SDK's frame normaliser accumulates a whole newline-free line into an unbounded `bytearray` before handing it over. That is now a docstring note and a docs sentence rather than a claim the backend cannot keep. I re-checked it against `opensandbox` 1.1.0 and `adapters/sse.py` is unchanged, so the caveat still stands on the current SDK.
Tests: `test_stalled_stream_destroys_the_sandbox_and_reports_a_timeout` and `test_stalled_stream_whose_sandbox_cannot_be_destroyed_still_reports_a_timeout`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
Review Comment:
You are right that `None` means "unparsable", not "dead". Now: an `error` with a prose `value` gives `exit_code=1` with the error text on stderr; no `error` and no `complete` gives `exit_code=-1` and a stderr note that no exit status was reported; neither is terminal. `timed_out` is `exit_code != 0 and elapsed >= timeout`, so a nonzero exit at the deadline reads as a timeout and a clean exit at the deadline does not.
You asked whether a real timeout and a signal kill had been exercised against the server. Both have now, against a local server:
```
sleep 60, budget 5s -> elapsed=6.0s exit_code=-1 timed_out=True sandbox_terminated=False stderr='signal: killed'
sh -c 'kill -9 $$', budget 30s -> elapsed=1.0s exit_code=-1 timed_out=False sandbox_terminated=False stderr='signal: killed'
```
The second is the case you were pointing at: execd words `error.value` as the prose `signal: killed`, `_infer_foreground_exit_code` returns `None` for it, and before this round that raised. It is now an ordinary failed command, and because it died 1.0s into a 30s budget it is not reported as a timeout either.
The fixtures now derive `exit_code` the way `_infer_foreground_exit_code` does (`_execution(error=..., complete=...)`), so the `-9`-with-no-`error` pair the SDK never returns is gone. Tests: `test_prose_error_value_is_a_failed_command_not_a_terminal_error`, `test_no_terminal_event_is_reported_not_terminal`, `test_nonzero_exit_at_the_deadline_is_a_timeout`, `test_clean_exit_at_the_deadline_is_not_a_timeout`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
Review Comment:
Done. On overflow the backend asks `files.get_file_info([path])` and reports `max(EntryInfo.size, bytes read)`, falling back to the bytes read when the lookup fails or reports 0, which a streaming source does. Tests: `test_oversized_read_stops_at_the_sentinel_byte_and_reports_the_real_size` (a 1,000,000-byte file is reported as such) and `test_oversized_read_never_reports_less_than_was_read`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
Review Comment:
Worth it, yes. `_get_connection_config` now refuses a connection with no `host` with a `SandboxTerminalError` that names the connection and the `opensandbox_conn_id=None` alternative, instead of letting `ConnectionConfigSync` fall back to `localhost:8080`. The `None` path, which the system test uses, is unchanged. Test: `test_connection_without_host_is_terminal_rather_than_localhost`, for both `None` and `""`; it also asserts no `ConnectionConfigSync` is built.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5611021776
<!-- uv-lock-rebase-notice -->
`uv.lock` on `main` just moved via [#72314](https://redirect.github.com/apache/airflow/pull/72314) ("[main] Upgrade important CI environment"), commit [`f044973`](https://redirect.github.com/apache/airflow/commit/f0449731644ba75474454a79c5720811a4f9ef4b) and this PR currently conflicts.
Quickest fix:
```bash
git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease
```
_Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future `uv.lock` bumps._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on code in PR #71676:
URL: https://github.com/apache/airflow/pull/71676#discussion_r4070144224
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
Review Comment:
Done -- `_get_sandbox` now runs before `started = time.monotonic()`. Test: `test_reconnect_time_is_not_charged_to_the_command`, where connect sleeps 0.3s against a 0.25s budget and an immediate signal kill is not reported as a timeout.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
+ except SandboxFileTooLargeError:
+ raise
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors("read a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+ finally:
+ close = getattr(chunks, "close", None)
+ if close is not None:
+ with suppress(Exception):
+ close()
+ return bytes(data)
+
+ def write_file(self, sandbox: str, path: str, content: bytes) -> None:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import WriteEntry
+
+ parent = posixpath.dirname(path)
+ if parent and parent != "/":
+ sandbox_client.files.create_directories([WriteEntry(path=parent, mode=755)])
+ sandbox_client.files.write_file(path, content, mode=644)
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"Could not write {path!r} in the sandbox.") from e
+ with _translate_opensandbox_errors("write a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+
+ def list_directory(self, sandbox: str, path: str) -> list[tuple[str, bool]]:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import DirectoryListEntry
+
+ entries = sandbox_client.files.list_directory(DirectoryListEntry(path=path, depth=1))
Review Comment:
Done. `_LIST_DIRECTORY_MAX_ENTRIES = 10_000`; above it `list_directory` raises a recoverable `SandboxError` telling the model to list a subdirectory or use `ls | head`. The comment is explicit about what the cap does and does not bound: the SDK has parsed the array by the time we see it, so this bounds what goes further, not the SDK's own peak. Test: `test_oversized_listing_is_refused_with_guidance`.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
+ except SandboxFileTooLargeError:
+ raise
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors("read a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+ finally:
+ close = getattr(chunks, "close", None)
+ if close is not None:
+ with suppress(Exception):
+ close()
+ return bytes(data)
+
+ def write_file(self, sandbox: str, path: str, content: bytes) -> None:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import WriteEntry
+
+ parent = posixpath.dirname(path)
+ if parent and parent != "/":
+ sandbox_client.files.create_directories([WriteEntry(path=parent, mode=755)])
+ sandbox_client.files.write_file(path, content, mode=644)
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"Could not write {path!r} in the sandbox.") from e
+ with _translate_opensandbox_errors("write a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+
+ def list_directory(self, sandbox: str, path: str) -> list[tuple[str, bool]]:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import DirectoryListEntry
+
+ entries = sandbox_client.files.list_directory(DirectoryListEntry(path=path, depth=1))
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors(
+ "list a sandbox directory", recoverable_statuses=frozenset({400})
+ ):
+ raise
+ return [
+ (posixpath.basename(entry.path.rstrip("/")), entry.entry_type == "directory") for entry in entries
+ ]
+
+ def destroy(self, sandbox: str) -> None:
+ sandbox_client = self._sandboxes.pop(sandbox, None)
+ try:
+ if sandbox_client is None:
+ from opensandbox import SandboxSync
+
+ sandbox_client = SandboxSync.connect(
Review Comment:
Done -- `destroy` reconnects with `skip_health_check=True`, while `_get_sandbox` on the run path keeps the readiness poll. Tests assert the kwarg on the destroy path (`test_uncached_sandbox_is_reconnected_without_a_readiness_poll`) and its absence on the run path.
---
Drafted-by: Claude Code (Opus 5); reviewed by @zozo123 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5661849001
<img width="1272" height="924" alt="opensandbox-airflow" src="https://github.com/user-attachments/assets/a51f16e2-62e9-4561-a08f-35f5734f257a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71676:
URL: https://github.com/apache/airflow/pull/71676#discussion_r3963682699
##########
providers/common/ai/docs/toolsets.rst:
##########
@@ -793,6 +793,56 @@ Constructor parameters:
guarantee this backend cannot make. Set ``"deny-all"`` after running
``sbx policy init deny-all``, or ``"allow-all"`` to state that egress is open.
+OpenSandbox backend
Review Comment:
Did you try it out? A working GIF or video would be ideal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #71676:
URL: https://github.com/apache/airflow/pull/71676#discussion_r4042195887
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
Review Comment:
The fail-closed guarantee here is the server's rather than ours. The SDK only forwards `networkPolicy` on the create request and documents nothing about refusing a policy it cannot enforce, so on a deployment without the egress sidecar this is a successful create of a sandbox with open egress, which is exactly the case `SandboxSpec` says a backend must raise on rather than provision something weaker ([base.py L85-88](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L85-L88)). The recording shows the matrix holding on a server that has the sidecar; did you get to try one that doesn't? `SandboxSync.get_egress_policy()` reads the enforced policy back off the sidecar, so a read-back after create (deny default, rules present) would make this ours to guarantee instead of a property of whichever server version someone deploys. Same for the docs sentence at `toolsets.rst` L842, which states
the rejection as fact.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
Review Comment:
Nothing bounds this call on our side. `RunCommandOpts(timeout=...)` is a server-side instruction, and the SDK streams over an SSE client built with `read=None` and a comment saying so ("SSE client (read timeout disabled)", `opensandbox/sync/adapters/command_adapter.py` L159-176), so `request_timeout` only covers connect and write. If execd stops sending events the call never returns, and the toolset awaits it in a thread it can't cancel, so a model-requested 60s command can hold a worker slot until the sandbox's own TTL (`sandbox_timeout`, default 3600) or indefinitely on a blackholed connection.
`SbxSandboxBackend` guards exactly this: it caps the subprocess at `timeout + _EXEC_GRACE` and returns `timed_out=True, sandbox_terminated=True` ([sbx.py L374-394](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/sbx.py#L374-L394)). `sandbox_terminated` is never set anywhere in this backend, so the toolset's "provision a fresh one" recovery is dead here.
Separately, `max_output_bytes` can't actually bound worker memory the way [base.py L180-183](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L180-L183) says it should. `_BoundedTail` holds to the cap for what it's handed, but the SSE layer materialises a whole output line before handing it over (the SDK's frame normaliser accumulates into an unbounded `bytearray` until a `\n\n` boundary, `opensandbox/adapters/sse.py` L36-60). One newline-free line, which model-written code can produce with `head -c 2G /dev/zero`, is fully resident first. Not fixable here without patching the SDK, but worth a docstring note since `sbx` is immune (64 KiB pipe reads, trimmed as it goes).
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
Review Comment:
Nothing tags the sandbox as Airflow's. `SandboxSync.create` takes `metadata: dict[str, str]`, `SandboxInfo` returns it and `SandboxFilter` can filter on it, so something like `{"created-by": "airflow"}` would let an operator on a shared server find these. `SbxSandboxBackend` already does this through `_new_sandbox_name()`'s `airflow-sandbox-` prefix, which [base.py L75-77](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L75-L77) describes as being there for correlation and cleanup. The server TTL covers reclamation, but not attribution.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
+ except SandboxFileTooLargeError:
+ raise
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors("read a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+ finally:
+ close = getattr(chunks, "close", None)
+ if close is not None:
+ with suppress(Exception):
+ close()
+ return bytes(data)
+
+ def write_file(self, sandbox: str, path: str, content: bytes) -> None:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import WriteEntry
+
+ parent = posixpath.dirname(path)
+ if parent and parent != "/":
+ sandbox_client.files.create_directories([WriteEntry(path=parent, mode=755)])
+ sandbox_client.files.write_file(path, content, mode=644)
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"Could not write {path!r} in the sandbox.") from e
+ with _translate_opensandbox_errors("write a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+
+ def list_directory(self, sandbox: str, path: str) -> list[tuple[str, bool]]:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import DirectoryListEntry
+
+ entries = sandbox_client.files.list_directory(DirectoryListEntry(path=path, depth=1))
Review Comment:
This override drops the cap the base class puts on guest-controlled output. The base runs `find` with `max_output_bytes=_FILE_OP_OUTPUT_CAP`, and [base.py L65-67](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L65-L67) gives the reason: "a small cap bounds what a hostile guest can push into worker memory". The native call has no equivalent and the SDK parses the whole JSON array into `FileInfo` objects eagerly, so a directory the model just filled with a few hundred thousand entries is fully resident before the toolset's `truncate_output` ever runs. `DirectoryListEntry` has no limit field, so this may need a cap on the returned list plus a note to the model that it was truncated.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
Review Comment:
`exit_code` isn't reported by the server on this path. For every foreground run the SDK synthesises it with `_infer_foreground_exit_code`, which returns `int(execution.error.value)`, `0`, or `None` (`opensandbox/sync/adapters/command_adapter.py` L68-76 and L215-216, reached unconditionally here because `background` is never set). `error.value` is free text, documented as "Error message explaining what went wrong".
So `None` means "the SDK couldn't parse an integer out of a free-text field", not "the sandbox is unusable", which is what [base.py L43-50](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L43-L50) reserves terminal for. I ran the inference directly: `'1'` gives 1, `'-9'` gives -9, and `'signal: killed'`, `'exit status 1'` and no-events-at-all all give `None`. Any execd build that words `error.value` as prose fails the task on an ordinary non-zero exit.
The timeout branch on L286 has the same root: it needs `error.value` to be a *negative integer string*, so `'124'` leaves `timed_out` False and the model sees `[exit code: 124]`, and prose lands on the raise above instead of the `[timed out after Ns]` the toolset documents as normal tool output. The fixtures can't catch it because they mock `commands.run` past the inference: `_execution(exit_code=-9)` leaves `error=None`, a pair the SDK never returns, and `test_execution_error_is_returned_on_stderr` sets `exit_code=1` with `error.value="failed"`, where the real `exit_code` would be `None` and this raise would fire before the stderr fallback that test is covering. Did a real timeout and a signal kill get exercised against the server?
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
Review Comment:
Worth rejecting an empty host? `request_timeout` and `use_server_proxy` just below both get strict validation, but a connection with no host leaves `domain` as `None`, and `ConnectionConfigSync` then falls back to `OPEN_SANDBOX_DOMAIN` and finally to `localhost:8080` (`opensandbox/config/connection_sync.py` L107), so a half-filled connection quietly points the worker at itself.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
+ except SandboxFileTooLargeError:
+ raise
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors("read a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+ finally:
+ close = getattr(chunks, "close", None)
+ if close is not None:
+ with suppress(Exception):
+ close()
+ return bytes(data)
+
+ def write_file(self, sandbox: str, path: str, content: bytes) -> None:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import WriteEntry
+
+ parent = posixpath.dirname(path)
+ if parent and parent != "/":
+ sandbox_client.files.create_directories([WriteEntry(path=parent, mode=755)])
+ sandbox_client.files.write_file(path, content, mode=644)
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"Could not write {path!r} in the sandbox.") from e
+ with _translate_opensandbox_errors("write a sandbox file", recoverable_statuses=frozenset({400})):
+ raise
+
+ def list_directory(self, sandbox: str, path: str) -> list[tuple[str, bool]]:
+ sandbox_client = self._get_sandbox(sandbox)
+ try:
+ from opensandbox.models.filesystem import DirectoryListEntry
+
+ entries = sandbox_client.files.list_directory(DirectoryListEntry(path=path, depth=1))
+ except Exception as e:
+ if _get_status_code(e) == 404:
+ self._confirm_sandbox_exists(sandbox_client)
+ raise SandboxError(f"{path!r} does not exist in the sandbox, or is not readable.") from e
+ with _translate_opensandbox_errors(
+ "list a sandbox directory", recoverable_statuses=frozenset({400})
+ ):
+ raise
+ return [
+ (posixpath.basename(entry.path.rstrip("/")), entry.entry_type == "directory") for entry in entries
+ ]
+
+ def destroy(self, sandbox: str) -> None:
+ sandbox_client = self._sandboxes.pop(sandbox, None)
+ try:
+ if sandbox_client is None:
+ from opensandbox import SandboxSync
+
+ sandbox_client = SandboxSync.connect(
Review Comment:
`connect` waits for readiness unless you pass `skip_health_check=True`, so this spends up to `ready_timeout` (120s) polling before it even tries to kill. For a sandbox that exists but isn't healthy, paused or with execd not answering, the poll then raises `SandboxReadyTimeoutException`, which carries no `status_code`, so the 404 check below can't fire and it becomes terminal. That's 120s spent to fail a method [base.py L287-289](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L287-L289) requires to be idempotent, and you don't need a healthy sandbox to destroy one. `skip_health_check=True` on this path looks right.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
+ with _translate_opensandbox_errors("run a sandbox command"):
+ from opensandbox.models.execd import RunCommandOpts
+ from opensandbox.models.execd_sync import ExecutionHandlersSync
+
+ execution = self._get_sandbox(sandbox).commands.run(
+ command,
+ opts=RunCommandOpts(timeout=timedelta(seconds=timeout)),
+ handlers=ExecutionHandlersSync(
+ on_stdout=stdout.add_message,
+ on_stderr=stderr.add_message,
+ skip_accumulation=True,
+ ),
+ )
+
+ if execution.exit_code is None:
+ raise SandboxTerminalError("OpenSandbox returned no terminal status for the command.")
+ if execution.error is not None and not stderr.get_text():
+ details = "\n".join(execution.error.traceback) or execution.error.value
+ stderr.add_text(details)
+ timed_out = execution.exit_code < 0 and time.monotonic() - started >= timeout
+ return SandboxExecResult(
+ exit_code=execution.exit_code,
+ stdout=stdout.get_text(),
+ stderr=stderr.get_text(),
+ timed_out=timed_out,
+ stdout_truncated=stdout.truncated,
+ stderr_truncated=stderr.truncated,
+ )
+
+ @staticmethod
+ def _confirm_sandbox_exists(sandbox: SandboxSync) -> None:
+ with _translate_opensandbox_errors("confirm that a sandbox still exists"):
+ sandbox.get_info()
+
+ def read_file(self, sandbox: str, path: str, *, max_bytes: int) -> bytes:
+ _validate_positive_finite(max_bytes, "max_bytes")
+ sandbox_client = self._get_sandbox(sandbox)
+ chunks = None
+ data = bytearray()
+ try:
+ chunks = sandbox_client.files.read_bytes_stream(
+ path,
+ chunk_size=min(65536, max_bytes + 1),
+ range_header=f"bytes=0-{max_bytes}",
+ )
+ for chunk in chunks:
+ data.extend(chunk[: max_bytes + 1 - len(data)])
+ if len(data) > max_bytes:
+ raise SandboxFileTooLargeError(path, len(data), max_bytes)
Review Comment:
`len(data)` is always `max_bytes + 1` here, so every oversized file is reported as one byte over the limit, and the toolset renders `size_bytes` straight into the message the model sees. At the default 5 MiB budget a 1 GB file comes back as "is 5.0MB, over the 5.0MB read limit", which isn't much to plan a slicing strategy from. The base implementation reports the `stat` size ([base.py L232-240](https://github.com/apache/airflow/blob/4eba473d6c0518e019b22adac4c913b3a723cb75/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py#L232-L240)), and `files.get_file_info([path])` returns `EntryInfo.size` if you want the same here.
##########
providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py:
##########
@@ -0,0 +1,381 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""OpenSandbox backend for :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`."""
+
+from __future__ import annotations
+
+import posixpath
+import time
+from contextlib import contextmanager, suppress
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.common.ai.sandbox.base import (
+ SandboxBackend,
+ SandboxError,
+ SandboxExecResult,
+ SandboxFileTooLargeError,
+ SandboxTerminalError,
+ _validate_positive_finite,
+)
+from airflow.providers.common.compat.sdk import BaseHook
+
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
+ from opensandbox import SandboxSync
+ from opensandbox.config import ConnectionConfigSync
+ from opensandbox.models.sandboxes import NetworkPolicy
+
+ from airflow.providers.common.ai.sandbox.base import SandboxSpec
+
+
+def _get_status_code(error: Exception) -> int | None:
+ status_code = getattr(error, "status_code", None)
+ return status_code if isinstance(status_code, int) else None
+
+
+@contextmanager
+def _translate_opensandbox_errors(
+ operation: str, *, recoverable_statuses: frozenset[int] = frozenset()
+) -> Iterator[None]:
+ try:
+ yield
+ except SandboxError:
+ raise
+ except Exception as e:
+ try:
+ from opensandbox.exceptions import SandboxApiException
+ except ImportError:
+ raise SandboxTerminalError(
+ "The OpenSandbox SDK is not installed. Install "
+ '"apache-airflow-providers-common-ai[sandbox-opensandbox]".'
+ ) from e
+ status_code = _get_status_code(e) if isinstance(e, SandboxApiException) else None
+ status = f" (HTTP {status_code})" if status_code is not None else ""
+ message = f"OpenSandbox could not {operation}{status}."
+ if status_code in recoverable_statuses:
+ raise SandboxError(message) from e
+ raise SandboxTerminalError(message) from e
+
+
+class _BoundedTail:
+ def __init__(self, max_bytes: int) -> None:
+ self._max_bytes = max_bytes
+ self._data = bytearray()
+ self.truncated = False
+
+ def add_text(self, text: str) -> None:
+ self._data.extend(text.encode("utf-8"))
+ if len(self._data) > self._max_bytes:
+ del self._data[: len(self._data) - self._max_bytes]
+ self.truncated = True
+
+ def add_message(self, message: Any) -> None:
+ # execd streams one message per output line with the delimiter stripped,
+ # so the newline has to be put back or every line runs together. A blank
+ # line already arrives as "\n", hence the guard.
+ text = message.text
+ self.add_text(text if text.endswith("\n") else text + "\n")
+
+ def get_text(self) -> str:
+ return bytes(self._data).decode("utf-8", errors="ignore")
+
+
+def _parse_bool(value: Any, name: str) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ normalized = value.strip().lower()
+ if normalized in {"true", "1", "yes"}:
+ return True
+ if normalized in {"false", "0", "no"}:
+ return False
+ raise SandboxTerminalError(f"The OpenSandbox connection extra {name} must be a boolean.")
+
+
+class OpenSandboxBackend(SandboxBackend):
+ """
+ Run sandbox tools through an OpenSandbox server.
+
+ OpenSandbox supports Docker and Kubernetes runtimes behind the same API.
+ Airflow workers need only network access to that API; the OpenSandbox
+ deployment owns container provisioning and isolation.
+
+ A generic Airflow connection supplies the server configuration. ``host``
+ and ``port`` identify the lifecycle API, ``schema`` selects ``http`` or
+ ``https``, and ``password`` carries the optional API key. Connection extras
+ may set ``request_timeout`` and ``use_server_proxy``.
+
+ Strict network policy requires the OpenSandbox egress sidecar. The server
+ rejects a requested policy when that component or runtime support is
+ unavailable, preserving :class:`~airflow.providers.common.ai.sandbox.SandboxSpec`'s
+ fail-closed contract.
+
+ :param opensandbox_conn_id: Generic Airflow connection ID. ``None`` lets the
+ SDK resolve ``OPEN_SANDBOX_DOMAIN`` and ``OPEN_SANDBOX_API_KEY``.
+ :param image: Container image used for each sandbox.
+ :param cpu: OpenSandbox CPU resource limit.
+ :param memory: OpenSandbox memory resource limit.
+ :param sandbox_timeout: Server-side sandbox lifetime in seconds.
+ :param ready_timeout: Seconds to wait for a newly created sandbox to become healthy.
+ :param use_server_proxy: Route sandbox service calls through the lifecycle
+ server. ``None`` reads the connection extra and otherwise defaults to ``True``.
+ """
+
+ name = "opensandbox"
+
+ def __init__(
+ self,
+ opensandbox_conn_id: str | None = "opensandbox_default",
+ *,
+ image: str = "python:3.12-slim",
+ cpu: str = "1",
+ memory: str = "2Gi",
+ sandbox_timeout: float = 3600.0,
+ ready_timeout: float = 120.0,
+ use_server_proxy: bool | None = None,
+ ) -> None:
+ if not image:
+ raise ValueError("image must not be empty.")
+ if not cpu:
+ raise ValueError("cpu must not be empty.")
+ if not memory:
+ raise ValueError("memory must not be empty.")
+ _validate_positive_finite(sandbox_timeout, "sandbox_timeout")
+ _validate_positive_finite(ready_timeout, "ready_timeout")
+ self._opensandbox_conn_id = opensandbox_conn_id
+ self._image = image
+ self._resource = {"cpu": cpu, "memory": memory}
+ self._sandbox_timeout = sandbox_timeout
+ self._ready_timeout = ready_timeout
+ self._use_server_proxy = use_server_proxy
+ self._connection_config: ConnectionConfigSync | None = None
+ self._sandboxes: dict[str, SandboxSync] = {}
+
+ def _get_connection_config(self) -> ConnectionConfigSync:
+ if self._connection_config is not None:
+ return self._connection_config
+ with _translate_opensandbox_errors("initialize its client"):
+ from opensandbox.config import ConnectionConfigSync
+
+ if self._opensandbox_conn_id is None:
+ self._connection_config = ConnectionConfigSync(
+ use_server_proxy=True if self._use_server_proxy is None else self._use_server_proxy
+ )
+ return self._connection_config
+
+ conn = BaseHook.get_connection(self._opensandbox_conn_id)
+ extra = conn.extra_dejson
+ request_timeout = extra.get("request_timeout", 30)
+ try:
+ request_timeout = float(request_timeout)
+ _validate_positive_finite(request_timeout, "connection extra request_timeout")
+ except (TypeError, ValueError) as e:
+ raise SandboxTerminalError(
+ "The OpenSandbox connection extra request_timeout must be a positive finite number."
+ ) from e
+
+ use_server_proxy = self._use_server_proxy
+ if use_server_proxy is None:
+ value = extra.get("use_server_proxy", True)
+ use_server_proxy = _parse_bool(value, "use_server_proxy")
+
+ domain = conn.host
+ if domain and conn.port:
+ domain = f"{domain}:{conn.port}"
+ self._connection_config = ConnectionConfigSync(
+ api_key=conn.password or None,
+ domain=domain,
+ protocol=conn.schema or "http",
+ request_timeout=timedelta(seconds=request_timeout),
+ use_server_proxy=use_server_proxy,
+ )
+ return self._connection_config
+
+ @staticmethod
+ def _get_network_policy(spec: SandboxSpec | None) -> NetworkPolicy | None:
+ if spec is None:
+ return None
+ if not spec.block_network and spec.allow_egress_to:
+ raise SandboxTerminalError(
+ "SandboxSpec.allow_egress_to only narrows a deny-by-default policy; "
+ "set block_network=True or remove the allowlist."
+ )
+ from opensandbox.models.sandboxes import NetworkPolicy, NetworkRule
+
+ rules = [NetworkRule(action="allow", target=target) for target in spec.allow_egress_to or ()]
+ # default_action is declared under its wire alias. populate_by_name means both
+ # spellings work at runtime, but only the alias is in the typed signature.
+ return NetworkPolicy(
+ defaultAction="deny" if spec.block_network else "allow",
+ egress=rules or None,
+ )
+
+ def create(self, *, spec: SandboxSpec | None = None) -> str:
+ with _translate_opensandbox_errors("create a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.create(
+ self._image,
+ timeout=timedelta(seconds=self._sandbox_timeout),
+ ready_timeout=timedelta(seconds=self._ready_timeout),
+ env=dict(spec.env) if spec is not None and spec.env else None,
+ resource=dict(self._resource),
+ network_policy=self._get_network_policy(spec),
+ connection_config=self._get_connection_config(),
+ )
+ self._sandboxes[sandbox.id] = sandbox
+ return sandbox.id
+
+ def _get_sandbox(self, sandbox_id: str) -> SandboxSync:
+ if sandbox := self._sandboxes.get(sandbox_id):
+ return sandbox
+ with _translate_opensandbox_errors("connect to a sandbox"):
+ from opensandbox import SandboxSync
+
+ sandbox = SandboxSync.connect(
+ sandbox_id,
+ connection_config=self._get_connection_config(),
+ connect_timeout=timedelta(seconds=self._ready_timeout),
+ )
+ self._sandboxes[sandbox_id] = sandbox
+ return sandbox
+
+ def run_command(
+ self, sandbox: str, command: str, *, timeout: float, max_output_bytes: int
+ ) -> SandboxExecResult:
+ _validate_positive_finite(timeout, "timeout")
+ _validate_positive_finite(max_output_bytes, "max_output_bytes")
+ stdout = _BoundedTail(max_output_bytes)
+ stderr = _BoundedTail(max_output_bytes)
+ started = time.monotonic()
Review Comment:
`started` is taken before the sandbox lookup on L271, and for an uncached handle `_get_sandbox` calls `SandboxSync.connect(connect_timeout=ready_timeout)`, which polls for readiness up to 120s by default. That reconnect time lands in the elapsed value feeding `timed_out` on L286, so a command that was signal-killed early after a slow reconnect gets reported to the model as having hit its budget. Moving this below the lookup fixes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenSandbox backend for sandbox tools [airflow]
Posted by "zozo123 (via GitHub)" <gi...@apache.org>.
zozo123 commented on PR #71676:
URL: https://github.com/apache/airflow/pull/71676#issuecomment-5677060560
Here’s the Airflow UI view from a fresh run of this PR’s `common_ai_sandbox_toolset_opensandbox` system-test DAG.
The manual run finished successfully in 16.4s. The task log shows the `python:3.12-slim` sandbox being created, passing its health check, torn down, and returning `sandbox boundary e2e passed`. The test exercises `write_file`, command execution, `read_file`, non-zero exit/stderr handling, directory listing/state persistence, and teardown.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org