You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/07/11 08:21:49 UTC
[PR] Docs: Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
potiuk opened a new pull request, #69750:
URL: https://github.com/apache/airflow/pull/69750
Adds graphviz-generated diagrams to the architecture overview
(core-concepts/overview.rst) explaining how a task actually runs across
Airflow's SDKs, in a new "Task execution architecture" section:
- Python Task SDK — the supervised path (Supervisor + forked task process over a
msgpack socket) vs the native in-process path (dag.test()), plus a numbered
message-flow sequence.
- Go SDK — standalone airflow-go-edge-worker pulling from the Edge API, running
the compiled bundle as a go-plugin gRPC subprocess and calling the Execution
API directly.
- Java (JVM) SDK — the new Coordinator layer (CoordinatorManager -> JavaCoordinator
-> _JavaActivitySubprocess) reusing the Python Supervisor, driving a JVM
subprocess over loopback TCP, with an architecture diagram and a numbered
execution-workflow sequence.
All diagrams are generated by the existing generate-airflow-diagrams prek hook
(committed as .py + .png + .md5sum).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69750:
URL: https://github.com/apache/airflow/pull/69750#issuecomment-5016549500
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/70100"><img src="https://img.shields.io/badge/PR-70100-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #69750:
URL: https://github.com/apache/airflow/pull/69750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #69750: Add Task SDK, Go and Java SDK execution architecture diagrams
URL: https://github.com/apache/airflow/pull/69750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs: Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #69750:
URL: https://github.com/apache/airflow/pull/69750#discussion_r3564403462
##########
airflow-core/docs/img/diagram_task_sdk_execution_sequence.py:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
Review Comment:
Done — this is now a true sequence diagram. Each participant (Executor, Task, Supervisor, Execution API) has its own lifeline, with the Supervisor placed in the middle so the Task ↔ Supervisor round-trip (steps 7→10: `GetConnection/GetVariable/GetXCom` → proxied Execution-API call → `*Result` back to the task) reads as arrows going back and forth between the two lifelines. Arrows are straight and the step descriptions are margined so they stay clear of the lines. Pushed in b5331899d5.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
##########
airflow-core/docs/img/diagram_java_sdk_execution_architecture.py:
##########
@@ -0,0 +1,287 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
+Architecture diagram for the Java (JVM) Task SDK.
+
+Unlike the Go SDK (a standalone edge worker), the Java SDK plugs into the *same*
+Python Supervisor via a new **Coordinator** layer:
+
+* ``CoordinatorManager`` resolves the task's ``queue`` to a ``BaseCoordinator``
+ (``JavaCoordinator`` for the ``java`` queue, ``_PythonCoordinator`` otherwise);
+* ``JavaCoordinator.execute_task()`` opens two loopback-TCP servers, spawns a JVM
+ bundle process with ``subprocess.Popen``, and drives it with
+ ``_JavaActivitySubprocess`` — a subclass of the shared ``ActivitySubprocess``;
+* the JVM process connects *back* over TCP and speaks the same msgpack protocol as
+ a Python task, so the Python side heartbeats, proxies every Execution-API call,
+ and manages state. The JVM task therefore **never holds the task JWT**.
+
+Rendered with graphviz directly so labels sit inside sized shapes: 3-D box =
+native OS process, rounded box = an object inside a process, component = a server
+app, cylinder = the database, note = a caption.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import graphviz
+from rich.console import Console
+
+MY_DIR = Path(__file__).parent
+MY_FILENAME = Path(__file__).with_suffix("").name
+
+console = Console(width=400, color_system="standard")
+
Review Comment:
Done — the Java execution flow (`diagram_java_sdk_execution_sequence.py`) is now a true sequence diagram: Scheduler, JVM subprocess, Supervisor and Execution API each on their own lifeline, Supervisor central, so the JVM ↔ Supervisor round-trip over loopback TCP (steps 9→12) shows as request/response arrows going back and forth. The architecture diagram stays as the static structure and the round-trip now lives in the sequence diagram. Pushed in b5331899d5.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
##########
airflow-core/docs/core-concepts/overview.rst:
##########
@@ -192,6 +192,88 @@ code is never executed in the context of the *scheduler*.
bundle version when dispatching each task. If needed, the cadence of sync and scan
of the *Dag bundle* can be configured.
+Task execution architecture
+---------------------------
+
+The diagrams above show how Airflow's components are *deployed*. The diagrams below instead show what happens
+*inside a worker when a task actually runs* — how the Task SDK, the Supervisor and Coordinator processes, and
+the language runtimes work together: which processes are involved, and the classes and protocols they use to
+communicate.
+
+.. _overview-task-sdk-execution-architecture:
+
+Python Task SDK execution
+.........................
+
+When a *worker* actually runs a task, it does not run the user's code directly. Instead it starts a
+lightweight **Supervisor** that runs in its own **native operating-system process** and
+*forks* a second native process in which the **Task SDK** runtime (``task_runner``) executes the user code.
+The two processes talk over a socket, and the Supervisor is the only side that ever holds the short-lived
+task JWT or talks to the *Execution API* — the user's code never sees the token and never touches the
+database.
+
+The same runtime can also run *in-process* (a single Python process, no fork, no sockets, no HTTP) for
+``dag.test()`` and local runs. The diagram below contrasts the two paths and marks where each Python process
+lives:
+
+.. image:: ../img/diagram_task_sdk_execution_architecture.png
+
+The message flow of a supervised run — startup, running the user code, proxied Connection/Variable/XCom
+lookups, heartbeats, and reporting the final state — is shown below as a numbered, top-to-bottom sequence.
+Each step names the process that performs it, and each arrow is coloured by its sender and labelled with the
+message class or protocol used:
+
+.. image:: ../img/diagram_task_sdk_execution_sequence.png
+
+.. _overview-non-python-language-sdks:
+
+Non-Python language SDKs (Go and Java)
+......................................
+
+The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic, so a task can also be written in
+a **compiled, non-Python language**. A Python Dag still declares the task with ``@task.stub(queue=...)`` (so
+Python and non-Python tasks can be mixed in one Dag), but the actual work is delegated to the matching runtime.
+There are currently **two different integration styles** — the Go SDK runs a standalone worker, while the Java
+SDK plugs into the existing Python Supervisor.
+
+.. _overview-go-sdk-architecture:
+
+**Go SDK — standalone edge worker.** The `Go Task SDK
Review Comment:
Sounds good — I've left the Go-SDK-first vs Coordinator-first ordering to your follow-up as you offered, since it's mainly your change and you can verify it yourself. Not touched in this update.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Docs: Add Task SDK, Go and Java SDK execution architecture diagrams [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69750:
URL: https://github.com/apache/airflow/pull/69750#discussion_r3564094902
##########
airflow-core/docs/core-concepts/overview.rst:
##########
@@ -192,6 +192,88 @@ code is never executed in the context of the *scheduler*.
bundle version when dispatching each task. If needed, the cadence of sync and scan
of the *Dag bundle* can be configured.
+Task execution architecture
+---------------------------
+
+The diagrams above show how Airflow's components are *deployed*. The diagrams below instead show what happens
+*inside a worker when a task actually runs* — how the Task SDK, the Supervisor and Coordinator processes, and
+the language runtimes work together: which processes are involved, and the classes and protocols they use to
+communicate.
+
+.. _overview-task-sdk-execution-architecture:
+
+Python Task SDK execution
+.........................
+
+When a *worker* actually runs a task, it does not run the user's code directly. Instead it starts a
+lightweight **Supervisor** that runs in its own **native operating-system process** and
+*forks* a second native process in which the **Task SDK** runtime (``task_runner``) executes the user code.
+The two processes talk over a socket, and the Supervisor is the only side that ever holds the short-lived
+task JWT or talks to the *Execution API* — the user's code never sees the token and never touches the
+database.
+
+The same runtime can also run *in-process* (a single Python process, no fork, no sockets, no HTTP) for
+``dag.test()`` and local runs. The diagram below contrasts the two paths and marks where each Python process
+lives:
+
+.. image:: ../img/diagram_task_sdk_execution_architecture.png
+
+The message flow of a supervised run — startup, running the user code, proxied Connection/Variable/XCom
+lookups, heartbeats, and reporting the final state — is shown below as a numbered, top-to-bottom sequence.
+Each step names the process that performs it, and each arrow is coloured by its sender and labelled with the
+message class or protocol used:
+
+.. image:: ../img/diagram_task_sdk_execution_sequence.png
+
+.. _overview-non-python-language-sdks:
+
+Non-Python language SDKs (Go and Java)
+......................................
+
+The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic, so a task can also be written in
+a **compiled, non-Python language**. A Python Dag still declares the task with ``@task.stub(queue=...)`` (so
+Python and non-Python tasks can be mixed in one Dag), but the actual work is delegated to the matching runtime.
+There are currently **two different integration styles** — the Go SDK runs a standalone worker, while the Java
+SDK plugs into the existing Python Supervisor.
+
+.. _overview-go-sdk-architecture:
+
+**Go SDK — standalone edge worker.** The `Go Task SDK
Review Comment:
For Go-SDK I prefer to mention the Coordinator approach first, then the edge worker one. Since the edge worker lack of the following feature this section pointed out -- https://github.com/apache/airflow/tree/main/go-sdk#known-limitations
Additionally, the Java, Go, upcoming TS SDK all go with the Coordinator as first-class direction.
I can do this in follow-up as those are mainly my change and I'm able to verify myself.
##########
airflow-core/docs/img/diagram_task_sdk_execution_sequence.py:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
Review Comment:
Ditto for the Task (Or perhaps task runner more accurately) <-> Supervisor part. How about having the round trip part as sequence diagram.
##########
airflow-core/docs/img/diagram_java_sdk_execution_architecture.py:
##########
@@ -0,0 +1,287 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
+Architecture diagram for the Java (JVM) Task SDK.
+
+Unlike the Go SDK (a standalone edge worker), the Java SDK plugs into the *same*
+Python Supervisor via a new **Coordinator** layer:
+
+* ``CoordinatorManager`` resolves the task's ``queue`` to a ``BaseCoordinator``
+ (``JavaCoordinator`` for the ``java`` queue, ``_PythonCoordinator`` otherwise);
+* ``JavaCoordinator.execute_task()`` opens two loopback-TCP servers, spawns a JVM
+ bundle process with ``subprocess.Popen``, and drives it with
+ ``_JavaActivitySubprocess`` — a subclass of the shared ``ActivitySubprocess``;
+* the JVM process connects *back* over TCP and speaks the same msgpack protocol as
+ a Python task, so the Python side heartbeats, proxies every Execution-API call,
+ and manages state. The JVM task therefore **never holds the task JWT**.
+
+Rendered with graphviz directly so labels sit inside sized shapes: 3-D box =
+native OS process, rounded box = an object inside a process, component = a server
+app, cylinder = the database, note = a caption.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import graphviz
+from rich.console import Console
+
+MY_DIR = Path(__file__).parent
+MY_FILENAME = Path(__file__).with_suffix("").name
+
+console = Console(width=400, color_system="standard")
+
Review Comment:
From user point of view, perhaps we could make the JVM <-> Supervisor part as sequence diagram to showcase the round-trip to make it more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org