You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ja...@apache.org on 2026/07/22 14:04:28 UTC
(airflow) branch main updated: Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk (#70156)
This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 38d94ba8188 Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk (#70156)
38d94ba8188 is described below
commit 38d94ba81889de9b804f0a6b1d227895b2ab9e0c
Author: Jason(Zhe-You) Liu <68...@users.noreply.github.com>
AuthorDate: Wed Jul 22 22:04:17 2026 +0800
Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk (#70156)
* Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk
* Unwrap ADR list line breaks in Lang-SDK ADR index
Manual line wrapping in the bullet list made the rendered Markdown
harder to scan; one line per entry reads better.
---
.../lang-sdk}/0001-java-sdk-airflow-integration.md | 0
.../adr/lang-sdk}/0002-workload-execution.md | 0
.../adr/lang-sdk}/0003-pure-java-dags.md | 0
.../adr/lang-sdk}/0004-dag-parsing.md | 0
.../adr/lang-sdk}/0005-coordinator-packaging.md | 0
.../lang-sdk}/0006-no-lang-sdk-source-display.md | 0
airflow-core/adr/lang-sdk/README.md | 37 ++++++++++++++++++++++
dev/breeze/tests/test_selective_checks.py | 14 --------
go-sdk/README.md | 3 ++
.../adr/0003-coordinator-protocol-msgpack-ipc.md | 14 ++++----
java-sdk/README.md | 4 +--
11 files changed, 49 insertions(+), 23 deletions(-)
diff --git a/java-sdk/adr/0001-java-sdk-airflow-integration.md b/airflow-core/adr/lang-sdk/0001-java-sdk-airflow-integration.md
similarity index 100%
rename from java-sdk/adr/0001-java-sdk-airflow-integration.md
rename to airflow-core/adr/lang-sdk/0001-java-sdk-airflow-integration.md
diff --git a/java-sdk/adr/0002-workload-execution.md b/airflow-core/adr/lang-sdk/0002-workload-execution.md
similarity index 100%
rename from java-sdk/adr/0002-workload-execution.md
rename to airflow-core/adr/lang-sdk/0002-workload-execution.md
diff --git a/java-sdk/adr/0003-pure-java-dags.md b/airflow-core/adr/lang-sdk/0003-pure-java-dags.md
similarity index 100%
rename from java-sdk/adr/0003-pure-java-dags.md
rename to airflow-core/adr/lang-sdk/0003-pure-java-dags.md
diff --git a/java-sdk/adr/0004-dag-parsing.md b/airflow-core/adr/lang-sdk/0004-dag-parsing.md
similarity index 100%
rename from java-sdk/adr/0004-dag-parsing.md
rename to airflow-core/adr/lang-sdk/0004-dag-parsing.md
diff --git a/java-sdk/adr/0005-coordinator-packaging.md b/airflow-core/adr/lang-sdk/0005-coordinator-packaging.md
similarity index 100%
rename from java-sdk/adr/0005-coordinator-packaging.md
rename to airflow-core/adr/lang-sdk/0005-coordinator-packaging.md
diff --git a/java-sdk/adr/0006-no-lang-sdk-source-display.md b/airflow-core/adr/lang-sdk/0006-no-lang-sdk-source-display.md
similarity index 100%
rename from java-sdk/adr/0006-no-lang-sdk-source-display.md
rename to airflow-core/adr/lang-sdk/0006-no-lang-sdk-source-display.md
diff --git a/airflow-core/adr/lang-sdk/README.md b/airflow-core/adr/lang-sdk/README.md
new file mode 100644
index 00000000000..7092bc19f31
--- /dev/null
+++ b/airflow-core/adr/lang-sdk/README.md
@@ -0,0 +1,37 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Architectural Decision Records — Language SDKs and coordinators
+
+These ADRs record the cross-cutting architecture decisions behind running non-Python tasks in
+Airflow ([AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ)): the coordinator layer, workload
+execution, packaging, and how Lang SDKs touch Airflow core surfaces (Dag parsing, `DagCode`, the
+REST API, and the UI). They originated in `java-sdk/adr/` and were moved here following the review
+discussion on [apache/airflow#70059](https://github.com/apache/airflow/pull/70059), because they
+bind core interfaces and apply to every language SDK, not just the Java SDK.
+
+- [ADR-0001](0001-java-sdk-airflow-integration.md): Java SDK / Airflow integration and the coordinator extension point.
+- [ADR-0002](0002-workload-execution.md): workload execution through coordinators.
+- [ADR-0003](0003-pure-java-dags.md): pure Java Dags — build-time packaging and code visibility.
+- [ADR-0004](0004-dag-parsing.md): language-specific Dag file processing.
+- [ADR-0005](0005-coordinator-packaging.md): coordinator packaging, module layout, and registration.
+- [ADR-0006](0006-no-lang-sdk-source-display.md): no Lang-SDK source display for mixed-language (`@task.stub`) Dags.
+
+Decisions specific to a single SDK stay next to that SDK — for example, the Go SDK's bundle-format
+decisions live in [`go-sdk/adr/`](../../../go-sdk/adr).
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index 196744ea2cd..8b215ba4dd8 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -1469,15 +1469,6 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
},
id="Skip java unit and e2e tests for java-sdk README-only change",
),
- pytest.param(
- ("java-sdk/adr/0001-java-sdk-airflow-integration.md",),
- {
- "run-java-sdk-tests": "false",
- "run-java-sdk-e2e-tests": "false",
- "prod-image-build": "false",
- },
- id="Skip java unit and e2e tests for java-sdk ADR-only change",
- ),
pytest.param(
("airflow-e2e-tests/docker/java.yml",),
{
@@ -1777,11 +1768,6 @@ def test_expected_output_pull_request_main(
True,
id="ktlint skipped when only java-sdk docs change",
),
- pytest.param(
- ("java-sdk/adr/0001-java-sdk-airflow-integration.md",),
- True,
- id="ktlint skipped when only java-sdk ADR docs change",
- ),
],
)
def test_ktlint_hook_only_runs_for_java_sdk_changes(files: tuple[str, ...], ktlint_skipped: bool):
diff --git a/go-sdk/README.md b/go-sdk/README.md
index 7bd6b3d13c8..a84373bd291 100644
--- a/go-sdk/README.md
+++ b/go-sdk/README.md
@@ -369,6 +369,9 @@ The [`adr/`](./adr) directory records the design decisions behind the SDK:
- [ADR 0004](./adr/0004-self-contained-executable-bundle.md): the self-contained executable bundle, where
the executable *is* the bundle.
+Cross-cutting Lang-SDK decisions — the coordinator architecture and how non-Python tasks integrate with
+Airflow core surfaces — are recorded in [`airflow-core/adr/lang-sdk/`](../airflow-core/adr/lang-sdk).
+
The normative, language-agnostic on-disk bundle format (the footer layout, manifest fields, and what the
`ExecutableCoordinator` reads) is specified in
[`executable-bundle-spec.rst`](https://github.com/apache/airflow/blob/main/task-sdk/docs/executable-bundle-spec.rst).
diff --git a/go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md b/go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md
index 82798bdeb10..a78c7bb92e4 100644
--- a/go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md
+++ b/go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md
@@ -56,9 +56,9 @@ local-process only.
Meanwhile, the Python side of Airflow has standardised on a different
wire protocol for non-Python language runtimes — the *coordinator
protocol* — pioneered by the Java SDK and described in
-[java-sdk ADR 0004](../../java-sdk/adr/0004-dag-parsing.md)
+[Lang-SDK ADR 0004](../../airflow-core/adr/lang-sdk/0004-dag-parsing.md)
and
-[java-sdk ADR 0002](../../java-sdk/adr/0002-workload-execution.md).
+[Lang-SDK ADR 0002](../../airflow-core/adr/lang-sdk/0002-workload-execution.md).
Its shape is:
- The runtime is launched with `--comm=<host:port>` and
@@ -153,7 +153,7 @@ When `Serve` enters coordinator mode it:
2. **Connects out** to the comm address, then to the logs address. Both
are TCP. We dial; we do not listen. The launcher already has both
listeners up before exec'ing the binary
- ([java-sdk ADR 0004, "What the Base Class Handles Automatically"](../../java-sdk/adr/0004-dag-parsing.md#what-the-base-class-handles-automatically)).
+ ([Lang-SDK ADR 0004, "What the Base Class Handles Automatically"](../../airflow-core/adr/lang-sdk/0004-dag-parsing.md#what-the-base-class-handles-automatically)).
3. **Routes structured logs to the logs socket.** A new
`slog.Handler` writes JSON-line records (one record per line, UTF-8,
@@ -165,7 +165,7 @@ When `Serve` enters coordinator mode it:
4. **Reads the first comm frame and dispatches by message type.** The
first frame's body has a `type` field per the Java SDK's encoding
- ([java-sdk ADR 0002, "Task SDK Protocol Messages"](../../java-sdk/adr/0002-workload-execution.md#task-sdk-protocol-messages)).
+ ([Lang-SDK ADR 0002, "Task SDK Protocol Messages"](../../airflow-core/adr/lang-sdk/0002-workload-execution.md#task-sdk-protocol-messages)).
Two values are valid here:
- `DagFileParseRequest` → DAG-parsing one-shot.
@@ -189,7 +189,7 @@ Supervisor Bundle binary (Go)
│ ├── serialise(reg) →
│ │ DagFileParsingResult
│ │ in DagSerialization v3 JSON
- │ │ (see java-sdk ADR-0004)
+ │ │ (see Lang-SDK ADR-0004)
│ │
│◄────────────────[4B len][msgpack: ────┤
│ id, {type: "DagFileParsingResult",
@@ -203,7 +203,7 @@ The serialised DAG payload must match Python's `SerializedDAG.serialize_dag`
output **exactly**, including the `__type` / `__var` wrapping rules,
unwrapping of "non-decorated" fields (`start_date`, `end_date`, `tags`),
and the timetable encoding listed in
-[java-sdk ADR 0004, "DagFileParsingResult Format"](../../java-sdk/adr/0004-dag-parsing.md#dagfileparsingresult-format).
+[Lang-SDK ADR 0004, "DagFileParsingResult Format"](../../airflow-core/adr/lang-sdk/0004-dag-parsing.md#dagfileparsingresult-format).
The Go SDK gains a `serde` package that performs this encoding from
`bundlev1.Bundle` / `bundlev1.Task`, validated against
`validation/serialization/test_dags.yaml` (the same fixture set the Java
@@ -269,7 +269,7 @@ this behaviour is identical in go-plugin mode and coordinator mode;
it is not a coordinator-only invention.
Frame correlation, error envelopes, and request `id` numbering follow
-java-sdk ADR 0002 verbatim. Re-implementing rather than reusing those
+Lang-SDK ADR 0002 verbatim. Re-implementing rather than reusing those
is a deliberate cost of having a separate Go runtime; the validation
fixtures keep the encoders honest.
diff --git a/java-sdk/README.md b/java-sdk/README.md
index e0108fb3d8d..7e3cbeafed1 100644
--- a/java-sdk/README.md
+++ b/java-sdk/README.md
@@ -590,7 +590,8 @@ The wire protocol is defined in
requires changes in **both** `schema.json` (Python side) and
`execution/Comm.kt` + `execution/Client.kt` (JVM side).
-See [Architectural Design Records](./adr) in the `adr` directory to learn more.
+See the [Architectural Decision Records](../airflow-core/adr/lang-sdk) in `airflow-core/adr/lang-sdk` to learn more — they
+cover the coordinator architecture and core integration surfaces shared by all language SDKs.
### Repository layout
@@ -609,7 +610,6 @@ java-sdk/
│ # calls to the Airflow log store
├── log4j2/ # Log4j 2 appender; routes Log4j 2 events to the Airflow log store
├── example/ # End-to-end example bundle (annotation + interface APIs, Java source)
-├── adr/ # Architectural Decision Records for the Java SDK
└── buildSrc/ # Shared Gradle convention plugins (Java version, lint, etc.)
```