You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/07/11 13:07:07 UTC
[PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
jason810496 opened a new pull request, #69757:
URL: https://github.com/apache/airflow/pull/69757
## Why
`@task.stub` tasks can only be declared argless today, so a Go task that needs an upstream's output has to hand-write `GetXCom` calls (with the upstream `task_id` hard-coded in Go, duplicating the wiring the Dag file already expresses). This PR makes the natural TaskFlow call work across the language boundary:
```python
@task.stub(queue="golang")
def transform(country: str, extracted: dict): ...
with DAG(...):
transform("uk", extract()) # extract() is a normal Python @task
```
```go
// The runtime binds "uk" onto country and pulls extract's XCom into extracted.
func transform(ctx sdk.TIRunContext, log *slog.Logger, country string, extracted map[string]any) error
```
## What
- **Parse time (providers/standard):** `_StubOperator` binds the TaskFlow call to the stub's signature and serializes an ordered positional-arg spec with the Dag (`_arg_bindings`): each entry is an XCom reference or a JSON literal, with `data_type` derived from the parameter annotation.
- **Server (execution API):** `ti_run` returns the spec as a new optional `TIRunContext.arg_bindings` field — extracted from the serialized-dag blob only for `_StubOperator` tasks, and stripped for older API versions (Cadwyn change in the in-progress `2026-06-30` block). The supervisor needs zero changes because `TIRunContext` is embedded verbatim in `StartupDetails`.
- **Supervisor schema:** new in-progress version `2026-07-30` (`AddArgBindingsToTIRunContext`); the downgrade path strips the field, so bundles pinned to `2026-06-16` keep working unchanged.
- **Go SDK:** new `pkg/binding` package. `Analyze` classifies parameters once at registration (injectables like `context.Context`/`*slog.Logger` vs JSON-decodable *data parameters*); `Resolve` binds the spec onto the data parameters in order — literals decode directly, XCom entries pull on demand. Arity or declared-type mismatches fail the task loudly before the body runs, replacing the old silent `reflect.Zero` fill.
- Fail-loud v1 exclusions at parse time: `.expand()`/`.partial()` (via a new generic `supports_expand` opt-out), `map`/`zip`/`concat` XComArgs, `*args`/`**kwargs`, context-key parameter names, and non-JSON-serializable literals.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3637137529
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
+ BOOLEAN = "boolean"
+ OBJECT = "object"
+ ARRAY = "array"
+ ANY = "any"
+
+
+class XComArgBinding(BaseModel):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ kind: Literal["xcom"]
Review Comment:
Nit?
```suggestion
kind: Literal["xcom"] = "xcom"
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
Review Comment:
JSON has no distinction between integer and number. Do we need both types, or is Number alone enough? (In go for example we already have to possibly/conditional convert Integer to different int types (int32, int, in64 etc etc)
##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -310,6 +332,30 @@ def ti_run(
should_retry=_is_eligible_to_retry(previous_state, ti.try_number, ti.max_tries),
)
+ # Only set for stub (foreign-runtime) tasks with a captured TaskFlow arg
+ # spec; the route excludes unset fields, keeping regular responses lean.
+ if ti.operator == _STUB_TASK_TYPE and (
+ arg_bindings := _get_arg_bindings(dag_bag, ti.dag_version_id, ti.task_id, session=session)
+ ):
+ try:
+ context.arg_bindings = get_arg_bindings_adapter().validate_python(arg_bindings)
+ except ValidationError:
+ log.exception(
+ "Serialized arg_bindings spec failed validation",
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
Review Comment:
```suggestion
task_id=ti.task_id,
dag_version_id=ti.dag_version_id,
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
Review Comment:
I wonder if we can reuse something from JSON Schema here instead of our own types...
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
@@ -3405,6 +3405,44 @@ def inner():
assert serialized3["python_callable_name"] == "empty_function"
+def test_stub_task_args_round_trip():
+ """The stub task's TaskFlow arg spec (``_arg_bindings``) survives Dag serialization."""
+ from airflow.sdk import task
+
+ with DAG(dag_id="arg_bindings_dag", schedule=None) as dag:
+
+ @task.stub
+ def extract(): ...
+
+ @task.stub
+ def transform(country: str, extracted: dict): ...
+
+ transform("uk", extract())
+
+ ser_dag = DagSerialization.to_dict(dag)
+ encoded_tasks = {t[Encoding.VAR]["task_id"]: t[Encoding.VAR] for t in ser_dag["dag"]["tasks"]}
+ assert "_arg_bindings" not in encoded_tasks["extract"], "argless stubs must not serialize a spec"
+ assert encoded_tasks["transform"]["_arg_bindings"] == [
+ {
+ Encoding.TYPE: DAT.DICT,
+ Encoding.VAR: {"name": "country", "kind": "literal", "data_type": "string", "value": "uk"},
+ },
+ {
+ Encoding.TYPE: DAT.DICT,
+ Encoding.VAR: {"name": "extracted", "kind": "xcom", "data_type": "object", "task_id": "extract"},
+ },
+ ]
+
+ round_tripped = DagSerialization.from_dict(ser_dag)
+ assert round_tripped.task_dict["transform"]._arg_bindings == [
+ {"name": "country", "kind": "literal", "data_type": "string", "value": "uk"},
+ {"name": "extracted", "kind": "xcom", "data_type": "object", "task_id": "extract"},
+ ]
+ assert not hasattr(round_tripped.task_dict["extract"], "_arg_bindings") or (
+ round_tripped.task_dict["extract"]._arg_bindings is None
+ )
Review Comment:
not hasattr or is None feels a bit odd. Pick one to assert.
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,22 +18,170 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
-from typing import TYPE_CHECKING, Any
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Union
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
DecoratedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+def _infer_data_type(annotation: Any) -> str:
+ """
+ Map a stub function parameter annotation to the language-neutral arg-type vocabulary.
+
+ The returned name is one of the execution API's ``ArgBindingDataType`` values
+ (``string``/``integer``/``number``/``boolean``/``object``/``array``/``any``); the foreign
+ runtime type-checks the bound value against it. Anything we cannot classify confidently
+ maps to ``any`` so binding falls back to a decode-only check.
+ """
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return "any"
+ origin = typing.get_origin(annotation)
+ if origin is not None:
+ if origin is Union or origin is types.UnionType:
+ members = [a for a in typing.get_args(annotation) if a is not type(None)]
+ if len(members) == 1:
+ return _infer_data_type(members[0])
+ return "any"
+ annotation = origin
+ if not isinstance(annotation, type):
+ return "any"
+ # bool subclasses int, and str/bytes are Sequences -- order matters.
+ if issubclass(annotation, bool):
+ return "boolean"
+ if issubclass(annotation, int):
+ return "integer"
+ if issubclass(annotation, float):
+ return "number"
+ if issubclass(annotation, str):
+ return "string"
+ if issubclass(annotation, bytes):
+ return "any"
+ if issubclass(annotation, (dict, Mapping)):
+ return "object"
+ if issubclass(annotation, (list, tuple, set, frozenset, Sequence)):
+ return "array"
+ return "any"
+
+
+def _build_arg_bindings(
+ python_callable: Callable,
+ op_args: Collection[Any],
+ op_kwargs: Mapping[str, Any],
+ task_id: str,
+) -> list[dict[str, Any]] | None:
+ """
+ Bind the TaskFlow call arguments to the stub signature and build the ordered arg spec.
+
+ Each spec entry is a plain dict matching one variant of the execution API's
+ ``TaskArgBinding`` union: an ``XComArgBinding`` (``kind="xcom"``) for upstream TaskFlow
+ outputs, or a ``LiteralArgBinding`` (``kind="literal"``) for everything else. ``name`` is
+ always the stub function's parameter name, so a foreign runtime can bind by name (e.g. the
+ Go SDK's ``sdk.TaskInput`` struct fields) in addition to the existing positional order.
+ Returns ``None`` for argless calls: the binding contract (including the signature checks
+ below) applies only once a TaskFlow call actually passes arguments, so pre-TaskFlow stub
+ Dags whose call arguments were always ignored keep parsing.
+ """
+ if not op_args and not op_kwargs:
+ return None
+
+ signature = inspect.signature(python_callable)
+
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+ bound = signature.bind(*op_args, **op_kwargs)
+ explicitly_bound = set(bound.arguments)
+ bound.apply_defaults()
+
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def get_annotation_for(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ spec: list[dict[str, Any]] = []
+ for name, param in signature.parameters.items():
+ value = bound.arguments[name]
+ data_type = _infer_data_type(get_annotation_for(name, param))
+ if isinstance(value, PlainXComArg):
+ if value.key != "return_value":
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
+ spec.append(
+ {
+ "name": name,
+ "kind": "xcom",
+ "data_type": data_type,
+ "task_id": value.operator.task_id,
+ }
+ )
+ continue
+ if isinstance(value, XComArg):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a "
+ f"{type(value).__name__}; only direct upstream task outputs can cross the "
+ "language boundary -- .map()/.zip()/.concat() results are not supported"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime"
+ )
+ entry: dict[str, Any] = {"name": name, "kind": "literal", "data_type": data_type, "value": value}
+ if name not in explicitly_bound:
+ entry["from_default"] = True
+ spec.append(entry)
+ return spec
+
+
class _StubOperator(DecoratedOperator):
custom_operator_name: str = "@task.stub"
+ # Mapped stubs would need per-map-index arg specs, which the foreign runtime cannot
Review Comment:
"Mapped stubs would need per-map-index arg specs"
I don't think this is true -- it's the same function for each mapped index (by design) so each index would receive the same type of arguments.
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
Review Comment:
What about Dates/DateTimes?
We handle those specially in xcom serialization already don't we?, and it's likely quite common to want to pass those along, so I think we should handle the "convert to target language native date type" rather than just passing it as a string and making task code parse it.
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -435,6 +436,13 @@ class TIRunContext(BaseModel):
always reflects when the task *first* started, not when it was rescheduled/resumed.
"""
+ arg_bindings: list[TaskArgBinding] | None = None
+ """
+ Ordered positional-argument binding spec for stub (foreign-runtime) tasks.
+
+ ``None`` for regular tasks and for stub tasks that declare no parameters.
+ """
Review Comment:
Hmmmmm, I wonder if this should not allow none, and make it an empty list in that case. I don't think it functionally makes a difference but... 🤔
##########
airflow-core/src/airflow/api_fastapi/execution_api/versions/__init__.py:
##########
@@ -51,9 +51,11 @@
AddTeamNameField,
AddVariableKeysEndpoint,
)
+from airflow.api_fastapi.execution_api.versions.v2026_07_30 import AddArgBindingsToTIRunContext
Review Comment:
What Airflow version are we targeting this for release? If its 3.4 then this should be further in the future, say 2026_10_30.
Or do you want to make a case that adding support for TaskFlow in go is a bug fix?
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
Review Comment:
Given how tightly integrated into the exec API this change is, I'm not sure putting it in standard provider is right -- my first thought is that this should live in/with the dag parsing code, not with the stub operator code?
##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -370,6 +371,87 @@ async def workload_token(request: Request) -> TIToken:
assert extras["scope"] == "execution"
assert extras["sub"] == str(ti.id)
+ def test_ti_run_returns_arg_bindings_for_stub_task(self, client, dag_maker):
+ """A stub task's TaskFlow arg spec is extracted from the serialized Dag and returned."""
+ with dag_maker("test_arg_bindings_dag", serialized=True):
+
+ @task.stub
+ def extract(): ...
+
+ @task.stub
+ def transform(country: str, extracted: dict, limit: int = 10): ...
+
+ transform("uk", extract())
+
+ dr = dag_maker.create_dagrun()
+ tis = {ti.task_id: ti for ti in dr.get_task_instances()}
+ for ti in tis.values():
+ ti.set_state(State.QUEUED)
+ dag_maker.session.flush()
+
+ payload = {
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-09-30T12:00:00Z",
+ }
+
+ response = client.patch(f"/execution/task-instances/{tis['transform'].id}/run", json=payload)
+ assert response.status_code == 200
+ assert response.json()["arg_bindings"] == [
+ {"name": "country", "kind": "literal", "data_type": "string", "value": "uk"},
+ {"name": "extracted", "kind": "xcom", "data_type": "object", "task_id": "extract"},
Review Comment:
data_type on an xcom arg feels slightly odd here -- we don't/cant know what the type is from the dag defn really, as it depends on what the task actually pushes doesn't it?
```python
@task
def upstream():
if CONDITION:
return {"data": "one"}
else
return False
```
Extreme case that Golang tasks may never be able to cope with, but regardless, we don't know the type of an xcom.
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
Review Comment:
I wonder if introducing arg_bindings should result in a bump in the serialization version? @amoghrajesh WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3729088376
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ if next(XComArg.iter_xcom_references(value), None) is not None:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a collection with an "
+ "upstream task output nested inside it; only a direct XComArg argument can cross "
+ "the language boundary -- pass the upstream output as its own argument"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
+ # isinstance, not .is_mapped: Airflow 2.11 operators have no is_mapped attribute.
+ if isinstance(value.operator, MappedOperator):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the aggregated "
+ f"output of the mapped task {value.operator.task_id!r}; a foreign runtime "
+ "pulls single XCom rows, so a mapped upstream's combined output is not "
+ "supported"
+ )
+ return True
+ if isinstance(value, XComArg):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a "
+ f"{type(value).__name__}; only direct upstream task outputs can cross the "
+ "language boundary -- .map()/.zip()/.concat() results are not supported"
Review Comment:
All the mapped operator cases will be supported in https://github.com/apache/airflow/pull/70570 as follow-up. It's a bit difficult but it's feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5127449392
<!-- uv-lock-rebase-notice -->
`uv.lock` on `main` just moved via commit [`b301429`](https://github.com/apache/airflow/commit/b3014293344d7e0fdce8bc62e9b6bb97c39baca2) and this PR currently conflicts.
Quickest fix:
```bash
git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease
```
_Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future `uv.lock` bumps._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728370605
##########
dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py:
##########
@@ -402,6 +402,39 @@ def _print_changes_table(changes_table):
console_print(syntax)
+def _resolve_existing_version_tag(version_tag: str) -> str:
Review Comment:
Why is this change in this PR?
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
Review Comment:
Stub operator should be made not-usable on Airflow2 if someone tries to instantiate one -- it wouldn't ever work anyway without the tasksdk machinery at execution time
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ if next(XComArg.iter_xcom_references(value), None) is not None:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a collection with an "
+ "upstream task output nested inside it; only a direct XComArg argument can cross "
+ "the language boundary -- pass the upstream output as its own argument"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
+ # isinstance, not .is_mapped: Airflow 2.11 operators have no is_mapped attribute.
+ if isinstance(value.operator, MappedOperator):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the aggregated "
+ f"output of the mapped task {value.operator.task_id!r}; a foreign runtime "
+ "pulls single XCom rows, so a mapped upstream's combined output is not "
+ "supported"
+ )
+ return True
+ if isinstance(value, XComArg):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a "
+ f"{type(value).__name__}; only direct upstream task outputs can cross the "
+ "language boundary -- .map()/.zip()/.concat() results are not supported"
Review Comment:
I can mayve see concat, and zip not being supported, but why no map?
Is this a "not for now" or a "it's really hard, possibly never"?
##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -372,6 +381,110 @@ async def workload_token(request: Request) -> TIToken:
assert extras["scope"] == "execution"
assert extras["sub"] == str(ti.id)
+ def test_ti_run_returns_arg_bindings_for_stub_task(self, client, dag_maker):
+ """A stub task's TaskFlow arg spec is extracted from the serialized Dag and returned."""
+ with dag_maker("test_arg_bindings_dag", serialized=True):
+
+ @task.stub
+ def extract(): ...
+
+ @task.stub
+ def transform(country: str, extracted: dict, limit: int = 10): ...
+
+ transform("uk", extract())
+
+ dr = dag_maker.create_dagrun()
+ tis = {ti.task_id: ti for ti in dr.get_task_instances()}
+ for ti in tis.values():
+ ti.set_state(State.QUEUED)
+ dag_maker.session.flush()
+
+ response = client.patch(f"/execution/task-instances/{tis['transform'].id}/run", json=self.RUN_PAYLOAD)
+ assert response.status_code == 200
+ assert response.json()["arg_bindings"] == [
+ {"name": "country", "kind": "literal", "value_schema": {"type": "string"}, "value": "uk"},
+ {
+ "name": "extracted",
+ "kind": "xcom",
+ "value_schema": {"type": "object", "additionalProperties": True},
Review Comment:
I'm still not sure what to make of the schema/data type on an xcom arg. How will it be used?
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ if next(XComArg.iter_xcom_references(value), None) is not None:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a collection with an "
+ "upstream task output nested inside it; only a direct XComArg argument can cross "
+ "the language boundary -- pass the upstream output as its own argument"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
Review Comment:
I don't think this limitation should stay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5264707957
Hi @ashb, please feel free to get any suggestion for follow-up when you have a moment, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3628531396
##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -310,6 +332,30 @@ def ti_run(
should_retry=_is_eligible_to_retry(previous_state, ti.try_number, ti.max_tries),
)
+ # Only set for stub (foreign-runtime) tasks with a captured TaskFlow arg
+ # spec; the route excludes unset fields, keeping regular responses lean.
+ if ti.operator == _STUB_TASK_TYPE and (
+ arg_bindings := _get_arg_bindings(dag_bag, ti.dag_version_id, ti.task_id, session=session)
+ ):
+ try:
+ context.arg_bindings = get_arg_bindings_adapter().validate_python(arg_bindings)
+ except ValidationError:
+ log.exception(
+ "Serialized arg_bindings spec failed validation",
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
+ )
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": (
+ "The serialized TaskFlow arg spec for this stub task is not valid on "
+ "this Airflow version; it may come from a newer providers release."
Review Comment:
```suggestion
"The serialized TaskFlow arg spec for this stub task is not valid.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5164922735
I can’t comment on the implementation, but the proposed interface looks very reasonable to me. It seems to me the described programming interface has not been fully implemented in this PR, and I don’t have enough knowledge to say whether this is on the right track or not tbh. Maybe it would be a good idea to edit the PR description to ground what exactly should be expected here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #69757:
URL: https://github.com/apache/airflow/pull/69757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728080264
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+
+Each binding is one variant of a union discriminated on ``kind``: an ``XComArgBinding``
+pulls the value from an upstream task's XCom, a ``LiteralArgBinding`` carries an inline
+value from the Dag file. Both variants still emit plain named structs
+(``$defs/XComArgBinding``, ``$defs/LiteralArgBinding``) for the foreign-language SDKs
+consuming the supervisor schema.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
+ BOOLEAN = "boolean"
+ OBJECT = "object"
+ ARRAY = "array"
+ ANY = "any"
+
+
+class XComArgBinding(BaseModel):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default on purpose: a required ``kind`` stays non-nullable through the OpenAPI
+ # round trip, which discriminated-union codegen needs (a defaulted field turns
+ # ``Literal`` into ``Literal | None`` in the generated task-sdk models).
+ kind: Literal["xcom"]
+
+ name: str
+ """The stub function's parameter name this binding fills, in declaration order."""
+
+ data_type: ArgBindingDataType = ArgBindingDataType.ANY
+ """Declared type from the stub function's annotation; runtimes type-check against it."""
+
+ task_id: str
+ """Upstream task id to pull the XCom from."""
+
+ key: str = "return_value"
+ """XCom key to pull."""
Review Comment:
That isn't true. This example from https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/taskflow.html
```python
@task
def get_ip():
return my_ip_service.get_main_ip()
@task(multiple_outputs=True)
def compose_email(external_ip):
return {
'subject':f'Server connected from {external_ip}',
'body': f'Your server executing Airflow is connected from the external IP {external_ip}<br>'
}
email_info = compose_email(get_ip())
EmailOperator(
task_id='send_email_notification',
to='example@example.com',
subject=email_info['subject'],
html_content=email_info['body']
)
```
shows that compose_email actually stores two different xcom_keys, `subject` and `body`, and `XcomArg`'s subscripte operator gives different keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3619530055
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+
+Each binding is one variant of a union discriminated on ``kind``: an ``XComArgBinding``
+pulls the value from an upstream task's XCom, a ``LiteralArgBinding`` carries an inline
+value from the Dag file. Both variants still emit plain named structs
+(``$defs/XComArgBinding``, ``$defs/LiteralArgBinding``) for the foreign-language SDKs
+consuming the supervisor schema.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
+ BOOLEAN = "boolean"
+ OBJECT = "object"
+ ARRAY = "array"
+ ANY = "any"
+
+
+class XComArgBinding(BaseModel):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default on purpose: a required ``kind`` stays non-nullable through the OpenAPI
+ # round trip, which discriminated-union codegen needs (a defaulted field turns
+ # ``Literal`` into ``Literal | None`` in the generated task-sdk models).
Review Comment:
```suggestion
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+
+Each binding is one variant of a union discriminated on ``kind``: an ``XComArgBinding``
+pulls the value from an upstream task's XCom, a ``LiteralArgBinding`` carries an inline
+value from the Dag file. Both variants still emit plain named structs
+(``$defs/XComArgBinding``, ``$defs/LiteralArgBinding``) for the foreign-language SDKs
+consuming the supervisor schema.
Review Comment:
```suggestion
```
##########
ts-sdk/src/generated/supervisor.ts:
##########
@@ -22,6 +22,20 @@
//
// Re-run with: pnpm run generate:supervisor
+/**
Review Comment:
We should not bump `ts-sdk/src/generated/supervisor.ts` in this PR.
##########
go-sdk/pkg/binding/binding.go:
##########
@@ -0,0 +1,701 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package binding turns a task function's parameter list into the concrete
+// argument values it is called with at execution time.
+//
+// Three kinds of parameter are supported:
+//
+// - Injectable runtime values: context.Context, sdk.TIRunContext,
+// *slog.Logger, and any interface whose method set is a subset of
+// sdk.Client. These are filled by type, in any position.
+// - Data parameters: everything else (except TaskInput structs, below), in
+// declaration order. They receive the positional arguments the Python
+// stub Dag captured at parse time from the TaskFlow call
+// (“transform("uk", extract())“) and delivered in StartupDetails. A
+// literal argument decodes directly; an XCom argument is pulled from the
+// named upstream task in the current dag run first.
+// - TaskInput structs: a struct that anonymously embeds sdk.TaskInput opts
+// into per-field, name-based binding instead of consuming one positional
+// slot as a whole-value decode target. Each exported field binds by name
+// against the Dag's TaskFlow call arguments: an `arg:"<name>"` tag names
+// the argument to claim, and a field with no tag claims its own Go field
+// name, verbatim. At most one such parameter is allowed per function.
+//
+// A TaskInput struct's fields are resolved first, by name, claiming entries
+// out of the argument spec; the remaining unclaimed entries are then
+// distributed, in their original relative order, onto the plain flat data
+// parameters in declaration order -- so flat parameters and a TaskInput
+// struct can coexist in the same function signature regardless of where each
+// sits, and (with no TaskInput struct present) this reduces to exactly
+// today's positional-only behaviour.
Review Comment:
We should make it either using flat-base or TaskInput struct-base. Not supporting flat-base plus TaskInput struct-base, which is too ambiguous.
##########
airflow-core/src/airflow/api_fastapi/execution_api/versions/v2026_06_30.py:
##########
@@ -141,3 +145,28 @@ def remove_partition_date_from_dag_run(response: ResponseInfo) -> None: # type:
"""Strip ``partition_date`` from the nested ``dag_run`` payload for older clients."""
if "dag_run" in response.body and isinstance(response.body["dag_run"], dict):
response.body["dag_run"].pop("partition_date", None)
+
+
+class AddArgBindingsToTIRunContext(VersionChange):
+ """
+ Add the ``arg_bindings`` positional-argument binding spec for stub (foreign-runtime) tasks.
+
+ Each entry is a discriminated union of ``XComArgBinding`` and ``LiteralArgBinding`` keyed
+ on ``kind``. ``data_type`` is declared as the ``ArgBindingDataType`` enum rather than an
+ inline ``Literal``; the wire representation (a JSON string) is unchanged, so no migration
+ instruction is needed -- this version has not been released with the ``arg_bindings`` field
+ in any other shape.
Review Comment:
```suggestion
```
##########
go-sdk/pkg/binding/binding.go:
##########
@@ -0,0 +1,701 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package binding turns a task function's parameter list into the concrete
+// argument values it is called with at execution time.
+//
+// Three kinds of parameter are supported:
+//
+// - Injectable runtime values: context.Context, sdk.TIRunContext,
+// *slog.Logger, and any interface whose method set is a subset of
+// sdk.Client. These are filled by type, in any position.
+// - Data parameters: everything else (except TaskInput structs, below), in
+// declaration order. They receive the positional arguments the Python
+// stub Dag captured at parse time from the TaskFlow call
+// (“transform("uk", extract())“) and delivered in StartupDetails. A
+// literal argument decodes directly; an XCom argument is pulled from the
+// named upstream task in the current dag run first.
+// - TaskInput structs: a struct that anonymously embeds sdk.TaskInput opts
+// into per-field, name-based binding instead of consuming one positional
+// slot as a whole-value decode target. Each exported field binds by name
+// against the Dag's TaskFlow call arguments: an `arg:"<name>"` tag names
+// the argument to claim, and a field with no tag claims its own Go field
+// name, verbatim. At most one such parameter is allowed per function.
+//
+// A TaskInput struct's fields are resolved first, by name, claiming entries
+// out of the argument spec; the remaining unclaimed entries are then
+// distributed, in their original relative order, onto the plain flat data
+// parameters in declaration order -- so flat parameters and a TaskInput
+// struct can coexist in the same function signature regardless of where each
+// sits, and (with no TaskInput struct present) this reduces to exactly
+// today's positional-only behaviour.
+//
+// Conceptually, flat data parameters are positional-argument binding: order
+// matters, and every parameter must be filled or Resolve fails the task
+// before its body runs. A TaskInput struct is closer to keyword-argument
+// binding: fields match by name, and a field whose name has no corresponding
+// TaskFlow call argument is simply left at its Go zero value instead of
+// failing the task -- the same way an unpassed keyword argument falls back
+// to its default in a kwargs-style call.
+//
+// Analyze inspects a function once at registration and returns a Plan; Resolve
+// builds the call arguments for each execution from that Plan and the
+// per-execution argument spec, failing loudly on arity or type mismatches of
+// flat data parameters. A declared type of "any" (or a Go parameter typed
+// any) opts that argument out of the type check; the decode step still fails
+// loudly on unusable values.
+//
+// Mapped upstream fan-in is out of scope: XCom arguments always pull the
+// unmapped upstream instance (map_index is never sent).
+package binding
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "log/slog"
+ "reflect"
+
+ "github.com/apache/airflow/go-sdk/pkg/api"
+ "github.com/apache/airflow/go-sdk/pkg/sdkcontext"
+ "github.com/apache/airflow/go-sdk/sdk"
+)
+
+// DataType is the language-neutral value type the Dag declared for an
+// argument (from the stub function's annotation on the Python side).
+type DataType string
+
+const (
+ DataTypeString DataType = "string"
+ DataTypeInteger DataType = "integer"
+ DataTypeNumber DataType = "number"
+ DataTypeBoolean DataType = "boolean"
+ DataTypeObject DataType = "object"
+ DataTypeArray DataType = "array"
+ DataTypeAny DataType = "any"
+)
Review Comment:
We should reference these from `go-sdk/pkg/execution/genmodels/models.gen.go` instead of hand-written to prevent drift
##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -110,6 +111,36 @@
log = structlog.get_logger(__name__)
tracer = trace.get_tracer(__name__)
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+_STUB_TASK_TYPE = "_StubOperator"
+
+
+def _get_arg_bindings(dag_version_id: UUID | None, task_id: str, *, session) -> list[dict] | None:
+ """Extract the stub task's serialized positional-arg spec from the serialized Dag blob."""
+ # Imported here on purpose: only the Multi-Lang stub-task path touches the
+ # serialized-dag machinery, so keep it off the module's top-level imports.
+ from airflow.models.dag_version import DagVersion
+ from airflow.serialization.enums import Encoding
+ from airflow.serialization.serialized_objects import BaseSerialization
+
+ if dag_version_id is None:
+ return None
+ dag_version = session.get(DagVersion, dag_version_id)
+ if dag_version is None or dag_version.serialized_dag is None:
+ return None
+ data = dag_version.serialized_dag.data
+ if not data:
Review Comment:
```suggestion
if not (data := dag_version.serialized_dag.data):
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+
+Each binding is one variant of a union discriminated on ``kind``: an ``XComArgBinding``
+pulls the value from an upstream task's XCom, a ``LiteralArgBinding`` carries an inline
+value from the Dag file. Both variants still emit plain named structs
+(``$defs/XComArgBinding``, ``$defs/LiteralArgBinding``) for the foreign-language SDKs
+consuming the supervisor schema.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
+ BOOLEAN = "boolean"
+ OBJECT = "object"
+ ARRAY = "array"
+ ANY = "any"
+
+
+class XComArgBinding(BaseModel):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default on purpose: a required ``kind`` stays non-nullable through the OpenAPI
+ # round trip, which discriminated-union codegen needs (a defaulted field turns
+ # ``Literal`` into ``Literal | None`` in the generated task-sdk models).
+ kind: Literal["xcom"]
+
+ name: str
+ """The stub function's parameter name this binding fills, in declaration order."""
+
+ data_type: ArgBindingDataType = ArgBindingDataType.ANY
+ """Declared type from the stub function's annotation; runtimes type-check against it."""
+
+ task_id: str
+ """Upstream task id to pull the XCom from."""
+
+ key: str = "return_value"
+ """XCom key to pull."""
Review Comment:
The `key` will always be `return_value` across TaskFlow, we can remove the `key` field from end to end.
##########
go-sdk/pkg/binding/binding.go:
##########
@@ -0,0 +1,701 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package binding turns a task function's parameter list into the concrete
+// argument values it is called with at execution time.
+//
+// Three kinds of parameter are supported:
+//
+// - Injectable runtime values: context.Context, sdk.TIRunContext,
+// *slog.Logger, and any interface whose method set is a subset of
+// sdk.Client. These are filled by type, in any position.
+// - Data parameters: everything else (except TaskInput structs, below), in
+// declaration order. They receive the positional arguments the Python
+// stub Dag captured at parse time from the TaskFlow call
+// (“transform("uk", extract())“) and delivered in StartupDetails. A
+// literal argument decodes directly; an XCom argument is pulled from the
+// named upstream task in the current dag run first.
+// - TaskInput structs: a struct that anonymously embeds sdk.TaskInput opts
+// into per-field, name-based binding instead of consuming one positional
+// slot as a whole-value decode target. Each exported field binds by name
+// against the Dag's TaskFlow call arguments: an `arg:"<name>"` tag names
+// the argument to claim, and a field with no tag claims its own Go field
+// name, verbatim. At most one such parameter is allowed per function.
+//
+// A TaskInput struct's fields are resolved first, by name, claiming entries
+// out of the argument spec; the remaining unclaimed entries are then
+// distributed, in their original relative order, onto the plain flat data
+// parameters in declaration order -- so flat parameters and a TaskInput
+// struct can coexist in the same function signature regardless of where each
+// sits, and (with no TaskInput struct present) this reduces to exactly
+// today's positional-only behaviour.
+//
+// Conceptually, flat data parameters are positional-argument binding: order
+// matters, and every parameter must be filled or Resolve fails the task
+// before its body runs. A TaskInput struct is closer to keyword-argument
+// binding: fields match by name, and a field whose name has no corresponding
+// TaskFlow call argument is simply left at its Go zero value instead of
+// failing the task -- the same way an unpassed keyword argument falls back
+// to its default in a kwargs-style call.
+//
+// Analyze inspects a function once at registration and returns a Plan; Resolve
+// builds the call arguments for each execution from that Plan and the
+// per-execution argument spec, failing loudly on arity or type mismatches of
+// flat data parameters. A declared type of "any" (or a Go parameter typed
+// any) opts that argument out of the type check; the decode step still fails
+// loudly on unusable values.
+//
+// Mapped upstream fan-in is out of scope: XCom arguments always pull the
+// unmapped upstream instance (map_index is never sent).
+package binding
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "log/slog"
+ "reflect"
+
+ "github.com/apache/airflow/go-sdk/pkg/api"
+ "github.com/apache/airflow/go-sdk/pkg/sdkcontext"
+ "github.com/apache/airflow/go-sdk/sdk"
+)
+
+// DataType is the language-neutral value type the Dag declared for an
+// argument (from the stub function's annotation on the Python side).
+type DataType string
+
+const (
+ DataTypeString DataType = "string"
+ DataTypeInteger DataType = "integer"
+ DataTypeNumber DataType = "number"
+ DataTypeBoolean DataType = "boolean"
+ DataTypeObject DataType = "object"
+ DataTypeArray DataType = "array"
+ DataTypeAny DataType = "any"
+)
+
+// Arg is one positional argument for a task function's data parameters, in
+// declaration order: an XComArg or a LiteralArg. A sealed sum type mirroring
+// the wire model's XComArgBinding/LiteralArgBinding split, kept runtime-neutral
+// so this package stays decoupled from the generated coordinator schema types.
+type Arg interface {
+ // ArgName is the stub function's parameter name this binding fills; used
+ // to match a TaskInput struct field's `arg:` tag (or its verbatim
+ // field-name fallback).
+ ArgName() string
+ // DeclaredType is the Dag-declared language-neutral type for the argument;
+ // empty is treated as DataTypeAny.
+ DeclaredType() DataType
+ // sealedArg restricts implementations to this package, keeping
+ // resolveOne's type switch the single exhaustive consumer.
+ sealedArg()
+}
+
+// XComArg sources the argument from an upstream task's XCom.
+type XComArg struct {
+ // Name is the stub function's parameter name this binding fills.
+ Name string
+ // TaskID is the upstream task to pull from.
+ TaskID string
+ // Key is the XCom key to pull; empty means the return-value key.
+ Key string
+ // DataType is the declared type to check the Go parameter against.
+ DataType DataType
+}
+
+// LiteralArg carries an inline value from the Dag file.
+type LiteralArg struct {
+ // Name is the stub function's parameter name this binding fills.
+ Name string
+ // Value is the literal value from the Dag file.
+ Value any
+ // DataType is the declared type to check the Go parameter against.
+ DataType DataType
+}
Review Comment:
We should reference these from go-sdk/pkg/execution/genmodels/models.gen.go instead of hand-written to prevent drift if possible.
##########
go-sdk/dags/go_examples.py:
##########
@@ -107,3 +119,90 @@ def concurrent_xcom_dag():
concurrent_xcom_dag()
+
+
+@task.stub(queue="golang")
+def make_config(): ...
+
+
+@task.stub(queue="golang")
+def make_numbers(): ...
+
+
+@task.stub(queue="golang")
+def via_flat_args(
+ name: str,
+ count: int,
+ ratio: float,
+ enabled: bool,
+ tags: list,
+ config: dict,
+ numbers: list,
+ note: str | None = None,
+): ...
+
+
+# Capitalized parameters on purpose: with no ``arg:`` tags on the Go side, each
+# struct field binds the argument spelled exactly like its Go field name.
+@task.stub(queue="golang")
+def via_struct_no_tags(RegionCode: str, Threshold: float): ...
+
+
+@task.stub(queue="golang")
+def via_struct_arg_tag(region_code: str, threshold: float): ...
+
+
+@task.stub(queue="golang")
+def via_struct_unmatched_arg(region_code: str): ...
+
+
+@dag(dag_id="taskflow_binding_dag")
+def taskflow_binding_dag():
+ """
+ Stress the TaskFlow argument-binding surface beyond ``simple_dag``'s transform.
+
+ Conceptually, the flat parameter list is *positional-argument* binding: order
+ matters, and every parameter must be filled or the task fails before it runs.
+ ``sdk.TaskInput`` structs are closer to *keyword-argument* binding: fields match
+ by name, and (see ``via_struct_unmatched_arg`` below) a field whose name has no
+ corresponding TaskFlow call argument simply stays at its zero value instead of
+ failing the task -- the same way an unpassed keyword argument falls back to a
+ default in kwargs-style calls.
+
+ ``via_flat_args``'s one mixed positional/keyword call carries literals of every
+ scalar type plus an array literal, and fans in XComs from *two* upstream Go
+ tasks: ``make_config`` returns an object that binds onto a strictly-decoded Go
+ struct, ``make_numbers`` an array that binds onto ``[]int``. ``note`` is not
+ passed, so its ``None`` default is captured and arrives in Go as a nil
+ ``*string``. The Go ``via_flat_args`` (``go-sdk/example/bundle/taskflowbinding``)
+ verifies every bound value and fails the task on any mismatch.
+
+ Three further tasks demonstrate the Go SDK's ``sdk.TaskInput`` struct injection
+ mode, one field-binding mode at a time:
+
+ * ``via_struct_no_tags``: no ``arg:`` tags at all -- each struct field binds
+ the argument spelled exactly like its Go field name, hence this stub's
+ capitalized ``RegionCode``/``Threshold`` parameters.
+ * ``via_struct_arg_tag``: every field names its argument via an explicit
+ ``arg:`` tag -- ``Region`` is genuinely renamed to ``region_code``, and
+ ``Threshold`` is tagged ``threshold`` to pull the snake_case argument its
+ verbatim field name would miss.
+ * ``via_struct_unmatched_arg``: the Go struct declares a field with no
+ corresponding argument in this TaskFlow call at all -- it stays at its Go
+ zero value rather than failing the task.
+ """
+ via_flat_args(
+ "summary",
+ 3,
+ 2.5,
+ True,
+ ["metrics", "hourly"],
+ config=make_config(),
+ numbers=make_numbers(),
+ )
+ via_struct_no_tags(RegionCode="eu-west-1", Threshold=0.75)
+ via_struct_arg_tag(region_code="eu-west-1", threshold=0.75)
+ via_struct_unmatched_arg(region_code="eu-west-1")
Review Comment:
For via_struct_no_tags, via_struct_arg_tag and via_struct_unmatched_arg tasks, we should make the parameters reference the value from XCom as well, not just exercising the literal case only. We need to make the changes e2e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3663929498
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task types (``TaskInstance.operator``, the operator class name) whose tasks carry a
+# lang-SDK ``arg_bindings`` spec. Used to gate the serialized-Dag lookup so regular tasks
+# never pay for it. The gate matches exact class names; a new lang-SDK operator adds its
+# name here.
+LANG_SDK_OPERATORS = frozenset({"_StubOperator"})
+
+
+def client_supports_arg_bindings() -> bool:
+ """
+ Whether the request's negotiated API version can receive ``arg_bindings``.
+
+ Clients on older versions never see the field (the version migration strips it from
+ the response), so the derivation must not run for them: a stub Dag that ran before
+ arg bindings existed keeps running against those clients. The version that introduced
+ the field is the single source of truth via Cadwyn's ``is_applied`` side-effect check.
+ """
Review Comment:
```suggestion
Clients on older versions never see the field (the version migration strips it from
the response), so the derivation must not run for them.
Additionally, we shouldn't distinguish by date, we should check the `VersionChangeWithSideEffects` subclass `.is_applied`.
Reference: https://docs.cadwyn.dev/concepts/version_changes/?h=versionchangewithsideeffects#version-changes-with-side-effects
"""
```
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,262 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead (e.g. datetimes, dates, "
+ "and times as ISO 8601 strings)"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
+ if value.operator.is_mapped:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the aggregated "
+ f"output of the mapped task {value.operator.task_id!r}; a foreign runtime "
+ "pulls single XCom rows, so a mapped upstream's combined output is not "
+ "supported"
+ )
+ return True
+ if isinstance(value, XComArg):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a "
+ f"{type(value).__name__}; only direct upstream task outputs can cross the "
+ "language boundary -- .map()/.zip()/.concat() results are not supported"
+ )
+ return False
+
+
+def _build_arg_bindings(
+ python_callable: Callable,
+ op_args: Collection[Any],
+ op_kwargs: Mapping[str, Any],
+ task_id: str,
+) -> list[dict[str, Any]] | None:
+ """
+ Bind the TaskFlow call arguments to the stub signature and build the ordered arg spec.
+
+ Each spec entry is a plain dict matching one variant of the execution API's
+ ``TaskArgBinding`` union: an ``XComArgBinding`` (``kind="xcom"``) for upstream TaskFlow
+ outputs, or a ``LiteralArgBinding`` (``kind="literal"``) for everything else. ``name`` is
+ always the stub function's parameter name, so a foreign runtime can bind by name (e.g. the
+ Go SDK's ``sdk.TaskInput`` struct fields) in addition to the existing positional order.
Review Comment:
```suggestion
always the stub function's parameter name, so a foreign runtime can bind by name in addition to the existing positional order.
```
##########
providers/standard/pyproject.toml:
##########
@@ -60,7 +60,8 @@ requires-python = ">=3.10"
# After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
dependencies = [
"apache-airflow>=2.11.0",
- "apache-airflow-providers-common-compat>=1.14.1",
+ # 1.18.0 first ships KNOWN_CONTEXT_KEYS and PlainXComArg, which the stub decorator imports.
+ "apache-airflow-providers-common-compat>=1.18.0", # use next version
Review Comment:
```suggestion
"apache-airflow-providers-common-compat>=1.18.0", # use next version
```
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,262 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead (e.g. datetimes, dates, "
+ "and times as ISO 8601 strings)"
+ )
Review Comment:
We handled the datetime and dates.
```suggestion
f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
"to the foreign runtime; pass it in its JSON form instead.
)
```
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -75,7 +318,30 @@ def __init__(
f"Functions passed to @task.stub must be an empty function (`pass`, or `...` only) (got {stmt})"
)
- ...
+ # Bind the TaskFlow call to the *original* signature (DecoratedOperator mangles context
+ # key defaults, which stubs reject anyway) and persist the ordered arg spec so the
+ # execution API can hand it to the foreign runtime via StartupDetails.
+ self._arg_bindings = _build_arg_bindings(python_callable, self.op_args, self.op_kwargs, self.task_id)
+
+ # Direct .expand() on the stub needs no parse-time spec (ti_run derives per-map-index
+ # bindings from the serialized expand input), but a mapped task group creates
+ # per-map-index instances of the tasks inside it with no expand input of their own,
+ # so their arg values are unresolvable both here and server-side.
+ in_mapped_group = self.get_closest_mapped_task_group() is not None
+ if self._arg_bindings is not None and in_mapped_group:
+ raise ValueError(
+ f"@task.stub task {self.task_id!r} passes TaskFlow call arguments inside a mapped "
+ "task group; the captured spec cannot carry values that resolve per map index at "
+ "runtime, so stub tasks with arguments are not supported under a task group's "
+ ".expand()"
+ )
Review Comment:
Should we move the `in_mapped_group = self.get_closest_mapped_task_group() is not None` validation into the `_build_arg_bindings`? Or at least before the `_build_arg_bindings` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5203944421
There’s also still a mention to sdk.TaskInput that does not exist. Otherwise I think this is good (aside from the comment above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3747619192
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from the ``@task.stub`` TaskFlow call, stored in the serialized
+Dag, and delivered to the lang-SDK runtime via ``TIRunContext.arg_bindings``.
+"""
+
+from __future__ import annotations
+
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+# A named, titled alias (like TaskArgBinding below) kept as free-form JSON rather than a
+# typed model, so unknown JSON-schema keywords survive re-serialization along the way.
+ArgValueSchema = TypeAliasType(
+ "ArgValueSchema", Annotated[dict[str, JsonValue], Field(title="ArgValueSchema")]
+)
+"""JSON-schema fragment constraining the value a stub-task argument binds to; generated
+by pydantic from the stub annotation, carried verbatim, unknown keywords ignored."""
+
+
+class _ArgBindingBase(BaseModel):
+ """Fields every :class:`TaskArgBinding` variant carries, regardless of ``kind``."""
+
+ name: str
+ """The stub function's parameter name this binding fills, in declaration order."""
+
+ value_schema: ArgValueSchema | None = None
+ """Schema fragment from the stub function's annotation; omitted when unconstrained."""
+
+
+class XComArgBinding(_ArgBindingBase):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default: it would drop ``kind`` from ``required``, and the generated task-sdk
+ # client then types it ``Literal | None``, invalid as a tagged-union discriminator.
+ kind: Literal["xcom"]
Review Comment:
I tried the following three approaches and would like to keep the current, as-is form to avoid corrupting the generated discriminated union.
`kind` is the discriminator (`Field(discriminator="kind")`) on the `TaskArgBinding`.
That union round-trips through `datamodel-code-generator`, which regenerates `task-sdk/src/airflow/sdk/api/datamodels/_generated.py` from this model's OpenAPI schema. All three options below were tested against the real generator (`uv run --active --group codegen --project apache-airflow-task-sdk --directory task-sdk -s dev/generate_task_sdk_models.py`).
#### 1. Plain default
```python
class XComArgBinding(_ArgBindingBase):
kind: Literal["xcom"] = "xcom"
```
`datamodel-code-generator` regenerates the client field with an added `| None`:
```diff
- kind: Annotated[Literal["xcom"], Field(title="Kind")]
+ kind: Annotated[Literal["xcom"] | None, Field(title="Kind")] = "xcom"
```
Because a default drops the field from the OpenAPI schema's `required` list, and the generator's response to "not required" is "wrap it in `Optional`" — regardless of the fact that the underlying type is a single-member `Literal` that **needs to stay bare for the discriminator to work**.
`airflow` itself, for example `airflow.sdk.exceptions` reaches `airflow.sdk`, which eagerly touches `_generated.py`now blows up at class-definition time:
```
File ".../task-sdk/src/airflow/sdk/api/datamodels/_generated.py", line 812, in <module>
class TaskArgBinding(RootModel[XComArgBinding | LiteralArgBinding]):
File ".../pydantic/_internal/_model_construction.py", line 256, in __new__
complete_model_class(
...
File ".../pydantic/_internal/_discriminated_union.py", line 473, in _infer_discriminator_values_for_inner_schema
raise PydanticUserError(
pydantic.errors.PydanticUserError: Model 'XComArgBinding' needs field 'kind' to be of type `Literal`
For further information visit https://errors.pydantic.dev/2.13/u/discriminator-needs-literal
```
#### 2. `Field(init=False, default=...)`
```python
class XComArgBinding(_ArgBindingBase):
kind: Literal["xcom"] = Field(init=False, default="xcom")
```
`init=False` only removes the field from `__init__`'s signature; it does nothing to whether the field is *required* for schema/validation purposes, which is what `datamodel-code-generator` and the discriminator care about.
```diff
- kind: Annotated[Literal["xcom"], Field(title="Kind")]
+ kind: Annotated[Literal["xcom"] | None, Field(title="Kind")] = "xcom"
```
Same `PydanticUserError: Model 'XComArgBinding' needs field 'kind' to be of type Literal` on import. `init=False` and a bare default are schema-equivalent as far as `required` is concerned.
#### 3. Bare `Literal`, no default (current, as-is)
```python
class XComArgBinding(_ArgBindingBase):
# No default: it would drop `kind` from `required`, and the generated task-sdk
# client then types it `Literal | None`, invalid as a tagged-union discriminator.
kind: Literal["xcom"]
```
`kind` stays in the OpenAPI schema's `required` list, so `datamodel-code-generator` emits it as a bare, non-optional `Literal`:
```python
kind: Annotated[Literal["xcom"], Field(title="Kind")]
```
`import airflow.sdk.api.datamodels._generated` succeeds cleanly. Every `TaskArgBinding` entry is built server-side as a plain dict literal in `build_arg_bindings` (`{"name": name, "kind": "xcom", ...}`), never via `XComArgBinding(...)`, so there's no call site that would benefit from a constructor-level default in the first place.
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -435,6 +436,13 @@ class TIRunContext(BaseModel):
always reflects when the task *first* started, not when it was rescheduled/resumed.
"""
+ arg_bindings: list[TaskArgBinding] | None = None
+ """
+ Ordered positional-argument binding spec for stub (foreign-runtime) tasks.
+
+ ``None`` for regular tasks and for stub tasks that declare no parameters.
+ """
Review Comment:
Sorry for leaving that hanging and I left it as `None` to keep the `None` semantic as "regular Python Task" and the argument-less `@task.stub` call.
The Lang SDK consumer side could even short circuit the further parsing based on the `None` semantic.
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
Review Comment:
The last time we bump -- https://github.com/apache/airflow/pull/55836/changes#diff-2f03f28d3201c630dfbae758caf95ada61ab10328de789fc464f6e27c4afb2d0 is for the Airflow 3 major release.
It seems we're unclear about the serialization versioning.
For example, should we use
- incremental versioning (just let it be `4` even though we're still in Airflow 3)
- semantic versioning (e.g. `v3_3_0_to_v3_3_4` ). Then should we also deal add `v3_3_1_to_v3_3_4` between each patch release?
- or follow the existing dated-base versioning like the execution API between each major version?
I tracked the issue in https://github.com/apache/airflow/issues/71364 as follow-up. I need some more design for this and perhaps raise out as a discussion (offline discussion first for sure).
##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -372,6 +381,110 @@ async def workload_token(request: Request) -> TIToken:
assert extras["scope"] == "execution"
assert extras["sub"] == str(ti.id)
+ def test_ti_run_returns_arg_bindings_for_stub_task(self, client, dag_maker):
+ """A stub task's TaskFlow arg spec is extracted from the serialized Dag and returned."""
+ with dag_maker("test_arg_bindings_dag", serialized=True):
+
+ @task.stub
+ def extract(): ...
+
+ @task.stub
+ def transform(country: str, extracted: dict, limit: int = 10): ...
+
+ transform("uk", extract())
+
+ dr = dag_maker.create_dagrun()
+ tis = {ti.task_id: ti for ti in dr.get_task_instances()}
+ for ti in tis.values():
+ ti.set_state(State.QUEUED)
+ dag_maker.session.flush()
+
+ response = client.patch(f"/execution/task-instances/{tis['transform'].id}/run", json=self.RUN_PAYLOAD)
+ assert response.status_code == 200
+ assert response.json()["arg_bindings"] == [
+ {"name": "country", "kind": "literal", "value_schema": {"type": "string"}, "value": "uk"},
+ {
+ "name": "extracted",
+ "kind": "xcom",
+ "value_schema": {"type": "object", "additionalProperties": True},
Review Comment:
Duplicate our offline discussion here:
> know "how to unpack the XCom" into native object
> > But that needs to be done at runtime after fetching the XCom -- we can't tell exactly what it might be at dag parse time.
Yes, I know. Even without the data type field, the SDK runtime can still parse into the native type. However, this means we treat the SDK runtime side annotation (kind of a client side) as source of truth for unpacking.
I prefer to keep the data type on an xcom arg with the following reasons:
1. Treat / unpack based on the server side (airflow core serialization) source of truth.
2. Make it possible to raise the early error when the type un-match on the Dag processing time (after AIP-85) for mixed Lang Dag
3. In the native Dag era, we can leverage this field to distinguish whether the upstream and downstream type is match or not in Dag processing time.
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ if next(XComArg.iter_xcom_references(value), None) is not None:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a collection with an "
+ "upstream task output nested inside it; only a direct XComArg argument can cross "
+ "the language boundary -- pass the upstream output as its own argument"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
Review Comment:
We only focus on the TaskFlow here and SDK runtime side even hard coded the `return_value` value. Or do you think both side should still handle the custom XCom key case?
I lean toward focusing on TaskFlow default `return_value` only.
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task types (``TaskInstance.operator``, the operator class name) whose tasks carry a
+# lang-SDK ``arg_bindings`` spec. Used to gate the serialized-Dag lookup so regular tasks
+# never pay for it. The gate matches exact class names; a new lang-SDK operator adds its
+# name here.
+LANG_SDK_OPERATORS = frozenset({"_StubOperator"})
Review Comment:
The whole implementation is migrated to core serialization layer, so it's now all "flag-based" instead of relying on the naming. (I plan to write an ADR for this design trade-off.)
```python
if (task := dag.task_dict.get(ti.task_id)) is None or not task.is_stub:
return None
return task.arg_bindings
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728104910
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -435,6 +436,13 @@ class TIRunContext(BaseModel):
always reflects when the task *first* started, not when it was rescheduled/resumed.
"""
+ arg_bindings: list[TaskArgBinding] | None = None
+ """
+ Ordered positional-argument binding spec for stub (foreign-runtime) tasks.
+
+ ``None`` for regular tasks and for stub tasks that declare no parameters.
+ """
Review Comment:
You resolved this but didn't change anything or say why it's fine as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728186774
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
Review Comment:
I don't think this needs a bump.
Its purely additive and only appears on `@task.stub` tasks that were called with taskflow args. `definitions.operator` also has `additionalProperties: true` and `validate_schema` only runs on the write path, so nothing on the read side changes for a v3 blob. `conversion_v3_to_v4` would probably be an empty function or very less.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3729070632
##########
dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py:
##########
@@ -402,6 +402,39 @@ def _print_changes_table(changes_table):
console_print(syntax)
+def _resolve_existing_version_tag(version_tag: str) -> str:
Review Comment:
My bad, this is a bad rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3679012379
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,277 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
Review Comment:
Addressed in https://github.com/apache/airflow/pull/69757/changes/748798de4de1f9d4829437652f8fb0b3a6c2937a. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3673479795
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,277 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
Review Comment:
Before the json.dumps, walk the value for embedded XCom references using the same helper that the regular TaskFlow uses..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728098254
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task types (``TaskInstance.operator``, the operator class name) whose tasks carry a
+# lang-SDK ``arg_bindings`` spec. Used to gate the serialized-Dag lookup so regular tasks
+# never pay for it. The gate matches exact class names; a new lang-SDK operator adds its
+# name here.
+LANG_SDK_OPERATORS = frozenset({"_StubOperator"})
Review Comment:
Yes, this is what we do for EmptyOperator -- we have a "is_empty" field or similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728203900
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
Review Comment:
I agree it's not "needed", no, but is it cleaner/a better forward pattern if we start making the version mean something?
In Airflow 2 the schema version was always 1, so it effectively never got used, and I'm not sure that's a good pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5263624510
> Do we need an ADR for this decisions made in this PR? Otherwise I think this is good.
Yes, I will add an ADR as follow-up to avoid the heavy full test CI run again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3643024932
##########
providers/standard/pyproject.toml:
##########
@@ -61,6 +61,8 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.14.1", # use next version
+ # The stub decorator generates arg-binding value schemas from parameter annotations
+ "pydantic>=2.11.0",
Review Comment:
`pydantic` is already part of the dependency in Airflow. I don't think we need to add them explicitly here (which might also cause further dependency conflict)
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -32,36 +32,21 @@
from airflow.api_fastapi.core_api.base import BaseModel
-# A named alias (like TaskArgBinding below) so both union branches of ArgValueSchema.type
-# reference one shared schema definition instead of two inlined enum copies; the explicit
-# title lets the supervisor-schema dump merge this def with the task-sdk-generated twin.
-JsonSchemaType = TypeAliasType(
- "JsonSchemaType",
- Annotated[
- Literal["string", "integer", "number", "boolean", "object", "array", "null"],
- Field(title="JsonSchemaType"),
- ],
+# A named alias (like TaskArgBinding below) so every schema keeps one shared, titled
Review Comment:
Shorting all these comments in this module we introduced as they're too verbose.
##########
providers/standard/tests/unit/standard/decorators/test_stub.py:
##########
@@ -282,51 +282,91 @@ def group(n):
pytest.param(bool, {"type": "boolean"}, id="bool"),
pytest.param(int, {"type": "integer", "format": "int64"}, id="int"),
pytest.param(float, {"type": "number", "format": "double"}, id="float"),
- pytest.param(dict, {"type": "object"}, id="dict"),
- pytest.param(dict[str, int], {"type": "object"}, id="dict-parameterized"),
- pytest.param(typing.Mapping[str, int], {"type": "object"}, id="mapping"),
- pytest.param(list, {"type": "array"}, id="list"),
- pytest.param(list[int], {"type": "array"}, id="list-parameterized"),
- pytest.param(tuple, {"type": "array"}, id="tuple"),
- pytest.param(set, {"type": "array"}, id="set"),
- pytest.param(typing.Sequence[int], {"type": "array"}, id="sequence"),
+ pytest.param(dict, {"type": "object", "additionalProperties": True}, id="dict"),
+ pytest.param(
+ dict[str, int],
+ {"type": "object", "additionalProperties": {"type": "integer", "format": "int64"}},
+ id="dict-parameterized",
+ ),
+ pytest.param(
+ typing.Mapping[str, int],
+ {"type": "object", "additionalProperties": {"type": "integer", "format": "int64"}},
+ id="mapping",
+ ),
+ pytest.param(list, {"type": "array", "items": {}}, id="list"),
+ pytest.param(
+ list[int],
+ {"type": "array", "items": {"type": "integer", "format": "int64"}},
+ id="list-parameterized",
+ ),
+ pytest.param(tuple, {"type": "array", "items": {}}, id="tuple"),
+ pytest.param(set, {"type": "array", "items": {}, "uniqueItems": True}, id="set"),
+ pytest.param(
+ typing.Sequence[int],
+ {"type": "array", "items": {"type": "integer", "format": "int64"}},
+ id="sequence",
+ ),
pytest.param(datetime.datetime, {"type": "string", "format": "date-time"}, id="datetime"),
- pytest.param(pendulum.DateTime, {"type": "string", "format": "date-time"}, id="pendulum-datetime"),
pytest.param(datetime.date, {"type": "string", "format": "date"}, id="date"),
pytest.param(datetime.time, {"type": "string", "format": "time"}, id="time"),
pytest.param(datetime.timedelta, {"type": "string", "format": "duration"}, id="timedelta"),
+ pytest.param(bytes, {"type": "string", "format": "binary"}, id="bytes"),
+ pytest.param(
+ typing.Literal["a", "b"],
+ {"type": "string", "enum": ["a", "b"]},
+ id="literal",
+ ),
pytest.param(Any, None, id="any"),
pytest.param(None, None, id="none"),
pytest.param(type(None), None, id="nonetype"),
- pytest.param(bytes, None, id="bytes"),
+ pytest.param(
+ pendulum.DateTime,
+ None,
+ id="pendulum-datetime",
+ # pydantic has no schema for arbitrary datetime subclasses; decode-only fallback.
+ ),
Review Comment:
The _ValueSchemaGenerator should handle the `pendulum.DateTime` as it's really common in Airflow.
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -39,84 +40,46 @@
from airflow.providers.common.compat.sdk import Context
-def _json_schema_fragment(annotation: Any) -> dict[str, Any] | None:
- """Map one non-union annotation to a JSON-schema fragment; ``None`` = unclassifiable."""
- if annotation is type(None):
- return {"type": "null"}
- origin = typing.get_origin(annotation)
- if origin is not None:
- annotation = origin
- if not isinstance(annotation, type):
- return None
- # bool subclasses int, str/bytes are Sequences, and datetime subclasses date -- order matters.
- if issubclass(annotation, bool):
- return {"type": "boolean"}
- if issubclass(annotation, int):
- return {"type": "integer", "format": "int64"}
- if issubclass(annotation, float):
- return {"type": "number", "format": "double"}
- if issubclass(annotation, str):
- return {"type": "string"}
- if issubclass(annotation, bytes):
- return None
- if issubclass(annotation, datetime.datetime):
- return {"type": "string", "format": "date-time"}
- if issubclass(annotation, datetime.date):
- return {"type": "string", "format": "date"}
- if issubclass(annotation, datetime.time):
- return {"type": "string", "format": "time"}
- if issubclass(annotation, datetime.timedelta):
- return {"type": "string", "format": "duration"}
- if issubclass(annotation, (dict, Mapping)):
- return {"type": "object"}
- if issubclass(annotation, (list, tuple, set, frozenset, Sequence)):
- return {"type": "array"}
- return None
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
"""
- Map a stub function parameter annotation to a JSON-schema fragment.
-
- Fragments carry the standard ``type`` keyword -- a single name, or a list for union
- annotations (set semantics; member order follows the annotation) -- plus a ``format``
- annotation where the Python type implies a wire representation the type name alone
- cannot (``int`` -> ``int64``, ``datetime`` -> ``date-time``, ``timedelta`` ->
- ``duration``, ...). Returns ``None`` when the annotation gives no constraint; the
- binding then omits ``value_schema`` and the foreign runtime falls back to a
- decode-only check.
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
Review Comment:
Short the docstring comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3658304314
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
+ if ti.map_index < 0:
+ _unsupported_arg_bindings("the task instance has not been expanded to a map index")
+
+ expand_value = expand_input.value
+ try:
+ sub_indexes = expand_input.resolve_expansion_sub_indexes(ti.map_index, ti.run_id, session=session)
+ except NotFullyPopulated as e:
+ _unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
+
+ spec = [
+ _bind_mapped_stub_arg(name, value, sub_index=None)
+ for name, value in (task.partial_kwargs.get("op_kwargs") or {}).items()
+ ]
+ spec += [
+ _bind_mapped_stub_arg(name, value, sub_index=sub_indexes[name])
+ for name, value in expand_value.items()
+ ]
+ return spec
+
+
+def _bind_mapped_stub_arg(name: str, value: Any, *, sub_index: int | None) -> dict[str, Any]:
Review Comment:
It's a large change. Defer to https://github.com/apache/airflow/issues/70523.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3654479476
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
Review Comment:
```suggestion
# TODO: Support the `expand_kwargs` path once #69757 merged and all the Lang-SDKs adapt it.
if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
_unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from the ``@task.stub`` TaskFlow call, stored in the serialized
+Dag, and delivered to the lang-SDK runtime via ``TIRunContext.arg_bindings``.
+"""
+
+from __future__ import annotations
+
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+# A named, titled alias (like TaskArgBinding below) kept as free-form JSON rather than a
+# typed model, so unknown JSON-schema keywords survive re-serialization along the way.
+ArgValueSchema = TypeAliasType(
+ "ArgValueSchema", Annotated[dict[str, JsonValue], Field(title="ArgValueSchema")]
+)
+"""JSON-schema fragment constraining the value a stub-task argument binds to; generated
+by pydantic from the stub annotation, carried verbatim, unknown keywords ignored."""
+
+
+class XComArgBinding(BaseModel):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default: it would drop ``kind`` from ``required``, and the generated task-sdk
+ # client then types it ``Literal | None``, invalid as a tagged-union discriminator.
+ kind: Literal["xcom"]
+
+ name: str
+ """The stub function's parameter name this binding fills, in declaration order."""
+
+ value_schema: ArgValueSchema | None = None
+ """Schema fragment from the stub function's annotation; omitted when unconstrained."""
+
+ task_id: str
+ """Upstream task id whose ``return_value`` XCom is pulled."""
+
+ map_index: int = -1
+ """Map index of the upstream XCom row to pull; -1 is the unmapped row."""
+
+ element_index: int | None = None
+ """When set, the pulled value is a sequence and this binding takes the element at
+ this index (the stub was expanded over an unmapped upstream's output)."""
Review Comment:
```suggestion
this index (the stub was expanded over an unmapped upstream's output). The Lang-SDK side will GET the single row (task_id, map_index=-1), decode it, take value[i]."""
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
+ if ti.map_index < 0:
+ _unsupported_arg_bindings("the task instance has not been expanded to a map index")
+
+ expand_value = expand_input.value
+ try:
+ sub_indexes = expand_input.resolve_expansion_sub_indexes(ti.map_index, ti.run_id, session=session)
+ except NotFullyPopulated as e:
+ _unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
Review Comment:
```suggestion
except NotFullyPopulated as e:
# In most of the happy path, this shouldn't happed at all unless someone clear upstream TIs or XCom themself during the DagRun.
_unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
+ if ti.map_index < 0:
+ _unsupported_arg_bindings("the task instance has not been expanded to a map index")
+
+ expand_value = expand_input.value
+ try:
+ sub_indexes = expand_input.resolve_expansion_sub_indexes(ti.map_index, ti.run_id, session=session)
+ except NotFullyPopulated as e:
+ _unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
+
+ spec = [
+ _bind_mapped_stub_arg(name, value, sub_index=None)
+ for name, value in (task.partial_kwargs.get("op_kwargs") or {}).items()
+ ]
+ spec += [
+ _bind_mapped_stub_arg(name, value, sub_index=sub_indexes[name])
+ for name, value in expand_value.items()
+ ]
+ return spec
+
+
+def _bind_mapped_stub_arg(name: str, value: Any, *, sub_index: int | None) -> dict[str, Any]:
+ """Build one arg-binding dict; ``sub_index`` is set for expanded kwargs, None for partial ones."""
+ if isinstance(value, SchedulerPlainXComArg):
+ if value.key != "return_value":
+ _unsupported_arg_bindings(f"parameter {name!r} references the XCom key {value.key!r}")
+ entry: dict[str, Any] = {"name": name, "kind": "xcom", "task_id": value.operator.task_id}
+ if sub_index is not None:
+ if value.operator.is_mapped:
+ entry["map_index"] = sub_index
+ else:
+ entry["element_index"] = sub_index
+ return entry
+ if isinstance(value, SchedulerXComArg):
+ _unsupported_arg_bindings(
+ f"parameter {name!r} received a {type(value).__name__}; only direct upstream"
+ " task outputs and literals are supported"
+ )
+ if sub_index is not None:
+ items = list(value.items()) if isinstance(value, dict) else value
Review Comment:
```suggestion
if sub_index is not None:
# this kwarg was expanded over a literal collection written in the Dag file
items = list(value.items()) if isinstance(value, dict) else value
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
+ if ti.map_index < 0:
+ _unsupported_arg_bindings("the task instance has not been expanded to a map index")
+
+ expand_value = expand_input.value
+ try:
+ sub_indexes = expand_input.resolve_expansion_sub_indexes(ti.map_index, ti.run_id, session=session)
+ except NotFullyPopulated as e:
+ _unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
+
+ spec = [
+ _bind_mapped_stub_arg(name, value, sub_index=None)
+ for name, value in (task.partial_kwargs.get("op_kwargs") or {}).items()
+ ]
+ spec += [
+ _bind_mapped_stub_arg(name, value, sub_index=sub_indexes[name])
+ for name, value in expand_value.items()
+ ]
+ return spec
+
+
+def _bind_mapped_stub_arg(name: str, value: Any, *, sub_index: int | None) -> dict[str, Any]:
Review Comment:
Would it be possible to set the `value_schema` (typing) as well?
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
+ """
+ Build the per-map-index arg spec for a mapped (``.expand()``) stub task.
+
+ A mapped stub never instantiates at parse time, so no spec is captured in the
+ serialized Dag; it is derived here from the serialized expand input instead, with
+ the map-index decomposition delegated to
+ ``SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes``.
+ Value schemas come from the stub function's annotations, which are not available
+ server-side, so mapped bindings omit them (runtimes fall back to decode-only checks).
+ """
+ expand_input = task._get_specified_expand_input()
+ if not isinstance(expand_input, SchedulerDictOfListsExpandInput):
+ _unsupported_arg_bindings("expand_kwargs() is not supported on stub tasks")
+ if ti.map_index < 0:
+ _unsupported_arg_bindings("the task instance has not been expanded to a map index")
+
+ expand_value = expand_input.value
+ try:
+ sub_indexes = expand_input.resolve_expansion_sub_indexes(ti.map_index, ti.run_id, session=session)
+ except NotFullyPopulated as e:
+ _unsupported_arg_bindings(f"upstream map lengths are not yet known for {sorted(e.missing)}")
+
+ spec = [
+ _bind_mapped_stub_arg(name, value, sub_index=None)
+ for name, value in (task.partial_kwargs.get("op_kwargs") or {}).items()
+ ]
+ spec += [
+ _bind_mapped_stub_arg(name, value, sub_index=sub_indexes[name])
+ for name, value in expand_value.items()
+ ]
Review Comment:
```suggestion
# partial() kwargs
spec = [
_bind_mapped_stub_arg(name, value, sub_index=None)
for name, value in (task.partial_kwargs.get("op_kwargs") or {}).items()
]
# expand() kwargs
spec += [
_bind_mapped_stub_arg(name, value, sub_index=sub_indexes[name])
for name, value in expand_value.items()
]
```
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,128 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Any, NoReturn
+
+from fastapi import HTTPException, status
+
+from airflow.models.expandinput import NotFullyPopulated, SchedulerDictOfListsExpandInput
+from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg, SchedulerXComArg
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task type recorded on the TI row (``TaskInstance.operator``) for
+# ``airflow.providers.standard.decorators.stub._StubOperator``. Used to gate the
+# serialized-Dag lookup for ``arg_bindings`` so regular tasks never pay for it.
+# The gate matches the exact class name; a subclass would need its own entry here.
+STUB_TASK_TYPE = "_StubOperator"
+
+
+def get_arg_bindings(dag_bag: DBDagBag, ti: Any, *, session: Session) -> list | None:
+ """Extract or derive the stub task's TaskFlow arg spec from its Dag version."""
+ if ti.dag_version_id is None:
+ return None
+ if (dag := dag_bag.get_dag(ti.dag_version_id, session=session)) is None:
+ return None
+ if (task := dag.task_dict.get(ti.task_id)) is None:
+ return None
+ if task.is_mapped:
+ return _resolve_mapped_stub_arg_bindings(task, ti, session=session)
+ return getattr(task, "_arg_bindings", None)
+
+
+def _unsupported_arg_bindings(detail: str) -> NoReturn:
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+ detail={
+ "reason": "invalid_arg_bindings",
+ "message": f"The stub task's TaskFlow arguments cannot be delivered: {detail}.",
+ },
+ )
+
+
+def _resolve_mapped_stub_arg_bindings(task: Any, ti: Any, *, session: Session) -> list[dict[str, Any]]:
Review Comment:
The task should be annotated with `Operator` where `Operator` is
```
Operator: TypeAlias = BaseOperator | MappedOperator
```
at `TYPE_CHECKING` level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Go SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3638459163
##########
airflow-core/src/airflow/api_fastapi/execution_api/versions/__init__.py:
##########
@@ -51,9 +51,11 @@
AddTeamNameField,
AddVariableKeysEndpoint,
)
+from airflow.api_fastapi.execution_api.versions.v2026_07_30 import AddArgBindingsToTIRunContext
Review Comment:
> What Airflow version are we targeting this for release? If its 3.4 then this should be further in the future, say 2026_10_30.
It's 3.4, I will make it 2026_10_30 then.
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,22 +18,170 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
-from typing import TYPE_CHECKING, Any
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Union
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
DecoratedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+def _infer_data_type(annotation: Any) -> str:
+ """
+ Map a stub function parameter annotation to the language-neutral arg-type vocabulary.
+
+ The returned name is one of the execution API's ``ArgBindingDataType`` values
+ (``string``/``integer``/``number``/``boolean``/``object``/``array``/``any``); the foreign
+ runtime type-checks the bound value against it. Anything we cannot classify confidently
+ maps to ``any`` so binding falls back to a decode-only check.
+ """
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return "any"
+ origin = typing.get_origin(annotation)
+ if origin is not None:
+ if origin is Union or origin is types.UnionType:
+ members = [a for a in typing.get_args(annotation) if a is not type(None)]
+ if len(members) == 1:
+ return _infer_data_type(members[0])
+ return "any"
+ annotation = origin
+ if not isinstance(annotation, type):
+ return "any"
+ # bool subclasses int, and str/bytes are Sequences -- order matters.
+ if issubclass(annotation, bool):
+ return "boolean"
+ if issubclass(annotation, int):
+ return "integer"
+ if issubclass(annotation, float):
+ return "number"
+ if issubclass(annotation, str):
+ return "string"
+ if issubclass(annotation, bytes):
+ return "any"
+ if issubclass(annotation, (dict, Mapping)):
+ return "object"
+ if issubclass(annotation, (list, tuple, set, frozenset, Sequence)):
+ return "array"
+ return "any"
+
+
+def _build_arg_bindings(
+ python_callable: Callable,
+ op_args: Collection[Any],
+ op_kwargs: Mapping[str, Any],
+ task_id: str,
+) -> list[dict[str, Any]] | None:
+ """
+ Bind the TaskFlow call arguments to the stub signature and build the ordered arg spec.
+
+ Each spec entry is a plain dict matching one variant of the execution API's
+ ``TaskArgBinding`` union: an ``XComArgBinding`` (``kind="xcom"``) for upstream TaskFlow
+ outputs, or a ``LiteralArgBinding`` (``kind="literal"``) for everything else. ``name`` is
+ always the stub function's parameter name, so a foreign runtime can bind by name (e.g. the
+ Go SDK's ``sdk.TaskInput`` struct fields) in addition to the existing positional order.
+ Returns ``None`` for argless calls: the binding contract (including the signature checks
+ below) applies only once a TaskFlow call actually passes arguments, so pre-TaskFlow stub
+ Dags whose call arguments were always ignored keep parsing.
+ """
+ if not op_args and not op_kwargs:
+ return None
+
+ signature = inspect.signature(python_callable)
+
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+ bound = signature.bind(*op_args, **op_kwargs)
+ explicitly_bound = set(bound.arguments)
+ bound.apply_defaults()
+
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def get_annotation_for(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ spec: list[dict[str, Any]] = []
+ for name, param in signature.parameters.items():
+ value = bound.arguments[name]
+ data_type = _infer_data_type(get_annotation_for(name, param))
+ if isinstance(value, PlainXComArg):
+ if value.key != "return_value":
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
+ spec.append(
+ {
+ "name": name,
+ "kind": "xcom",
+ "data_type": data_type,
+ "task_id": value.operator.task_id,
+ }
+ )
+ continue
+ if isinstance(value, XComArg):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a "
+ f"{type(value).__name__}; only direct upstream task outputs can cross the "
+ "language boundary -- .map()/.zip()/.concat() results are not supported"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime"
+ )
+ entry: dict[str, Any] = {"name": name, "kind": "literal", "data_type": data_type, "value": value}
+ if name not in explicitly_bound:
+ entry["from_default"] = True
+ spec.append(entry)
+ return spec
+
+
class _StubOperator(DecoratedOperator):
custom_operator_name: str = "@task.stub"
+ # Mapped stubs would need per-map-index arg specs, which the foreign runtime cannot
Review Comment:
I will double check this part.
##########
airflow-core/tests/unit/serialization/test_dag_serialization.py:
##########
Review Comment:
I will update the `airflow-core/src/airflow/serialization/schema.json` to show the new `args_binding` field (the native Dag TaskFlow will leverage the `args_binding` field as well). However, I don't think it's not necessary to bump the "serialization version".
##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -370,6 +371,87 @@ async def workload_token(request: Request) -> TIToken:
assert extras["scope"] == "execution"
assert extras["sub"] == str(ti.id)
+ def test_ti_run_returns_arg_bindings_for_stub_task(self, client, dag_maker):
+ """A stub task's TaskFlow arg spec is extracted from the serialized Dag and returned."""
+ with dag_maker("test_arg_bindings_dag", serialized=True):
+
+ @task.stub
+ def extract(): ...
+
+ @task.stub
+ def transform(country: str, extracted: dict, limit: int = 10): ...
+
+ transform("uk", extract())
+
+ dr = dag_maker.create_dagrun()
+ tis = {ti.task_id: ti for ti in dr.get_task_instances()}
+ for ti in tis.values():
+ ti.set_state(State.QUEUED)
+ dag_maker.session.flush()
+
+ payload = {
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-09-30T12:00:00Z",
+ }
+
+ response = client.patch(f"/execution/task-instances/{tis['transform'].id}/run", json=payload)
+ assert response.status_code == 200
+ assert response.json()["arg_bindings"] == [
+ {"name": "country", "kind": "literal", "data_type": "string", "value": "uk"},
+ {"name": "extracted", "kind": "xcom", "data_type": "object", "task_id": "extract"},
Review Comment:
Actually, all the "data_type" here only comes from the **annotation** at the Stub Operator level. So what exactly the upstream task return doesn't really matter IMO.
There comes up another case that I needs to resolve. The case you give, user might annotate the argument as `dict | bool`, but I haven't deal with the union type annotation yet.
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
Review Comment:
I will look into whether it would be better to change the dag parsing side or to keep it as is.
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from a stub task's TaskFlow call (``@task.stub``), stored in the
+serialized Dag, and delivered to the lang-SDK runtime through ``TIRunContext.arg_bindings``
+so it can bind the values onto the native task function's parameters.
+"""
+
+from __future__ import annotations
+
+from enum import Enum
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+
+class ArgBindingDataType(str, Enum):
+ """Language-neutral value type a stub-task argument binds to in the foreign runtime."""
+
+ STRING = "string"
+ INTEGER = "integer"
+ NUMBER = "number"
Review Comment:
Good call. I will make these JSON schema to make the "type" distinguishable across language.
https://json-schema.org/understanding-json-schema/reference/type#format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728087993
##########
airflow-core/src/airflow/api_fastapi/execution_api/services/task_instances.py:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Business logic backing the task-instance execution routes."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
+ from airflow.models.dagbag import DBDagBag
+
+# Task types (``TaskInstance.operator``, the operator class name) whose tasks carry a
+# lang-SDK ``arg_bindings`` spec. Used to gate the serialized-Dag lookup so regular tasks
+# never pay for it. The gate matches exact class names; a new lang-SDK operator adds its
+# name here.
+LANG_SDK_OPERATORS = frozenset({"_StubOperator"})
Review Comment:
Gating with operator type name is not a good idea. Add a flag on _StubOperator, serialize it, and read it here instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3728052497
##########
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/task_arg_binding.py:
##########
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Positional-argument binding spec for stub (foreign-runtime) tasks.
+
+Captured at parse time from the ``@task.stub`` TaskFlow call, stored in the serialized
+Dag, and delivered to the lang-SDK runtime via ``TIRunContext.arg_bindings``.
+"""
+
+from __future__ import annotations
+
+from functools import cache
+from typing import Annotated, Literal
+
+from pydantic import Field, JsonValue, TypeAdapter
+from typing_extensions import TypeAliasType
+
+from airflow.api_fastapi.core_api.base import BaseModel
+
+# A named, titled alias (like TaskArgBinding below) kept as free-form JSON rather than a
+# typed model, so unknown JSON-schema keywords survive re-serialization along the way.
+ArgValueSchema = TypeAliasType(
+ "ArgValueSchema", Annotated[dict[str, JsonValue], Field(title="ArgValueSchema")]
+)
+"""JSON-schema fragment constraining the value a stub-task argument binds to; generated
+by pydantic from the stub annotation, carried verbatim, unknown keywords ignored."""
+
+
+class _ArgBindingBase(BaseModel):
+ """Fields every :class:`TaskArgBinding` variant carries, regardless of ``kind``."""
+
+ name: str
+ """The stub function's parameter name this binding fills, in declaration order."""
+
+ value_schema: ArgValueSchema | None = None
+ """Schema fragment from the stub function's annotation; omitted when unconstrained."""
+
+
+class XComArgBinding(_ArgBindingBase):
+ """One positional stub-task argument pulled from an upstream task's XCom."""
+
+ # No default: it would drop ``kind`` from ``required``, and the generated task-sdk
+ # client then types it ``Literal | None``, invalid as a tagged-union discriminator.
+ kind: Literal["xcom"]
Review Comment:
Elsewhere we do this:
```python
state: Annotated[
Literal[TIState.RUNNING],
# Specify a default in the schema, but not in code.
WithJsonSchema({"type": "string", "enum": [TIState.RUNNING], "default": TIState.RUNNING}),
]
```
It might be worth seeing if we can do the same thing here to apply a default too?
Or this approach from ExecuteTask workload:
```python
type: Literal["ExecuteTask"] = Field(init=False, default="ExecuteTask")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #69757:
URL: https://github.com/apache/airflow/pull/69757#issuecomment-5204045324
I'm still not convinced about all this type handling logic being in the standard provider. A lot of this feels like it belongs in the dag parser, keeping the stub very simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support TaskFlow call syntax on stub tasks for the Lang SDK [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69757:
URL: https://github.com/apache/airflow/pull/69757#discussion_r3747921286
##########
providers/standard/src/airflow/providers/standard/decorators/stub.py:
##########
@@ -18,19 +18,283 @@
from __future__ import annotations
import ast
-from collections.abc import Callable
+import copy
+import datetime
+import inspect
+import json
+import types
+import typing
+from collections.abc import Callable, Collection, Mapping
+from functools import cache
from typing import TYPE_CHECKING, Any
+try:
+ from pydantic import PydanticUserError, TypeAdapter
+ from pydantic.json_schema import GenerateJsonSchema
+except ImportError:
+ # Airflow 3 always ships pydantic but Airflow 2.x base installs do not; without it,
+ # stub args carry no value schemas and runtimes keep their decode-only fallback.
+ GenerateJsonSchema = object # type: ignore[assignment,misc]
+ TypeAdapter = None # type: ignore[assignment,misc]
+ PydanticUserError = None # type: ignore[assignment,misc]
+
from airflow.providers.common.compat.sdk import (
+ KNOWN_CONTEXT_KEYS,
+ XCOM_RETURN_KEY,
DecoratedOperator,
+ MappedOperator,
+ PlainXComArg,
TaskDecorator,
+ XComArg,
task_decorator_factory,
)
if TYPE_CHECKING:
from airflow.providers.common.compat.sdk import Context
+class _ValueSchemaGenerator(GenerateJsonSchema):
+ """
+ Pydantic's stock JSON-schema generation plus OpenAPI's fixed-width numeric formats.
+
+ A foreign runtime decodes numbers into machine types, which the bare
+ ``integer``/``number`` type names cannot convey; ``format`` is an annotation per
+ JSON schema, so runtimes that don't know these names simply skip them.
+ """
+
+ def int_schema(self, schema):
+ return {**super().int_schema(schema), "format": "int64"}
+
+ def float_schema(self, schema):
+ return {**super().float_schema(schema), "format": "double"}
+
+
+# Most-derived first: datetime subclasses date, so it must be matched before date.
+_TEMPORAL_BASES = (datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
+
+
+def _normalize_temporal_annotation(annotation: Any) -> Any:
+ """
+ Map temporal subclasses (e.g. ``pendulum.DateTime``) to their stdlib base.
+
+ Applied recursively through unions and containers, and only as a retry when direct
+ schema generation fails, so temporal types carrying their own pydantic schema keep it.
+ """
+ # Parametrized generics must be detected before the plain-class branch: on Python
+ # 3.10, isinstance(list[X], type) is True and issubclass silently consults the
+ # origin, so the class branch would return list[X] unnormalized.
+ origin = typing.get_origin(annotation)
+ args = typing.get_args(annotation)
+ if origin is not None and args:
+ normalized = tuple(_normalize_temporal_annotation(arg) for arg in args)
+ if normalized == args:
+ return annotation
+ if origin in (typing.Union, types.UnionType):
+ return typing.Union[normalized] # noqa: UP007 -- runtime construction from a tuple
+ return origin[normalized]
+ if isinstance(annotation, type):
+ return next((base for base in _TEMPORAL_BASES if issubclass(annotation, base)), annotation)
+ return annotation
+
+
+def _infer_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Build the JSON-schema fragment for one stub parameter annotation, via pydantic.
+
+ The pydantic-generated schema ships verbatim, so runtimes must treat it as
+ open-vocabulary JSON schema. Returns ``None`` when the annotation constrains nothing
+ (missing, ``Any``, bare ``None``) or pydantic cannot generate a schema for it; the
+ binding then omits ``value_schema`` and the foreign runtime falls back to a
+ decode-only check.
+ """
+ if TypeAdapter is None:
+ return None
+ if annotation is inspect.Parameter.empty or annotation is None or annotation is Any:
+ return None
+ if annotation is type(None):
+ # get_type_hints normalizes a bare ``None`` annotation to NoneType; a parameter
+ # that can only ever be None constrains nothing worth shipping.
+ return None
+ try:
+ schema = _generate_value_schema(annotation)
+ except TypeError:
+ # Unhashable annotations cannot key the cache; generate directly. Any pydantic
+ # failure inside the body degrades to None there, so this retry never re-raises.
+ schema = _generate_value_schema.__wrapped__(annotation)
+ # Deep-copy so callers embedding the fragment never alias the cached dict.
+ return copy.deepcopy(schema) if schema else None
+
+
+@cache
+def _generate_value_schema(annotation: Any) -> dict[str, Any] | None:
+ """
+ Generate the schema for one annotation, cached for the process lifetime.
+
+ TypeAdapter construction is one of pydantic's most expensive operations and
+ annotations are static, so re-parses of the same Dag file must not re-pay it.
+ """
+ # Reached only when pydantic is installed (``_infer_value_schema`` guards on
+ # ``TypeAdapter is None``), so ``PydanticUserError`` is a real exception class here.
+ # It is the base of PydanticSchemaGenerationError and PydanticInvalidForJsonSchema and
+ # covers annotations pydantic rejects outright (e.g. bare ClassVar); TypeError catches
+ # the exotic generics pydantic chokes on with a plain TypeError. Either way, "pydantic
+ # cannot schema this" degrades to no schema rather than failing Dag parsing.
+ try:
+ return TypeAdapter(annotation).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ normalized = _normalize_temporal_annotation(annotation)
+ if normalized is annotation:
+ return None
+ try:
+ return TypeAdapter(normalized).json_schema(schema_generator=_ValueSchemaGenerator)
+ except (PydanticUserError, TypeError):
+ return None
+
+
+def _validate_stub_signature(signature: inspect.Signature, task_id: str) -> None:
+ for param in signature.parameters.values():
+ if param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
+ raise ValueError(
+ f"@task.stub task {task_id!r} must declare a fixed number of parameters for the "
+ f"foreign runtime to bind against; *{param.name} is not supported"
+ )
+ if param.name in KNOWN_CONTEXT_KEYS:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {param.name!r} is an Airflow context key; "
+ "stub signatures declare only data parameters -- the lang-SDK runtime injects its "
+ "own task context natively (e.g. the Go SDK's sdk.TIRunContext parameter)"
+ )
+
+
+def _resolve_param_annotations(python_callable: Callable, signature: inspect.Signature) -> dict[str, Any]:
+ """Map each parameter to its parse-time-resolvable annotation (``Parameter.empty`` when not)."""
+ try:
+ hints = typing.get_type_hints(python_callable)
+ except (NameError, TypeError):
+ # Annotations that cannot be resolved at parse time (e.g. names behind
+ # TYPE_CHECKING with ``from __future__ import annotations``) degrade to "any".
+ hints = {}
+
+ def resolve(name: str, param: inspect.Parameter) -> Any:
+ if name in hints:
+ return hints[name]
+ if isinstance(param.annotation, str):
+ return inspect.Parameter.empty
+ return param.annotation
+
+ return {name: resolve(name, param) for name, param in signature.parameters.items()}
+
+
+def _ensure_json_literal(value: Any, task_id: str, name: str) -> None:
+ if next(XComArg.iter_xcom_references(value), None) is not None:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a collection with an "
+ "upstream task output nested inside it; only a direct XComArg argument can cross "
+ "the language boundary -- pass the upstream output as its own argument"
+ )
+ try:
+ json.dumps(value, allow_nan=False)
+ except (TypeError, ValueError):
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} received a literal of type "
+ f"{type(value).__name__} that is not JSON-serializable, so it cannot be passed "
+ "to the foreign runtime; pass it in its JSON form instead"
+ )
+
+
+def _validate_xcom_value(value: Any, task_id: str, name: str) -> bool:
+ """Validate an XComArg argument, returning True when it is a bindable direct upstream output."""
+ if isinstance(value, PlainXComArg):
+ if value.key != XCOM_RETURN_KEY:
+ raise ValueError(
+ f"@task.stub task {task_id!r} parameter {name!r} references the XCom key "
+ f"{value.key!r}; only an upstream task's return value can cross the language "
+ "boundary -- indexing an output by a custom key is not supported"
+ )
Review Comment:
We only focus on the TaskFlow here and SDK runtime side even hard coded the `return_value` value. Or do you think both side should still handle the custom XCom key case? (Java SDK even remove the custom key support before the beta release as we think it unnecessary to support.)
I lean toward focusing on TaskFlow default `return_value` only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org