You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pierrejeambrun (via GitHub)" <gi...@apache.org> on 2026/09/30 09:31:57 UTC
[PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
pierrejeambrun opened a new pull request, #73948:
URL: https://github.com/apache/airflow/pull/73948
Records the decision behind apache/airflow#73723 (merged): a native Lang-SDK
bundle embeds each Dag-defining source file, de-duplicated by path, always
embeds the entry file as a fallback, and carries no Lang-SDK source for a
mixed-language Dag (Python owns its Code tab).
This settles the source-display question [ADR-0010](https://github.com/apache/airflow/blob/main/airflow-core/adr/lang-sdk/0010-native-dag-processing.md)
left open for native Dags ("what `get_source_code` returns … is not settled
here"), and keeps it consistent with [ADR-0006](https://github.com/apache/airflow/blob/main/airflow-core/adr/lang-sdk/0006-no-lang-sdk-source-display.md)
(mixed-language Dags show only their Python file). It also states the cross-SDK
convention — Go and Java follow the same shape (per-Dag attribution, dedup by
path, entrypoint fallback) — per the review follow-up on #73723.
New ADR: `airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md`,
plus its index entry in the lang-sdk ADR README. The "Relationship to ADR-0014"
section flags the one open alignment with the proposed metadata-slimming ADR
(which states "no `dag_id` anywhere"), to be resolved before either is Accepted.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143538152
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
Review Comment:
I just removed the Dag term and trimmed the last statement which is only targeted to native Dag case but mentioning here will be confusing.
```suggestion
The bundle only supplies task handlers for is owned by Python, which owns its Code tab
([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
returns nothing — never the entrypoint — so the caller shows the Python file.
```
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
Review Comment:
```suggestion
### 4. Mixed-language Tasks carry no Lang-SDK source
```
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
Not sure would this be more clear?
```suggestion
| native, the common case includes loop/factory-generated Dags | that file's source |
| native, with external condition to generate the Dags (e.g. read a external YAML file to render the Dags) | the entrypoint source (fallback) |
| mixed-language (Python-owned) | nothing — Python owns the Code tab |
```
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
Review Comment:
```suggestion
| the Task is… | reader returns |
```
I think it's fine to use "Task" term here as the Code tab represents the relationship of TaskInstance -> DagModel -> DagCode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143890826
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
External conditions are fine, as long as the condition is met a 'built' time.
That's also a requirement for having the dag at all in the output bundle (condition for running it).
(for instance if it's data driven and when running the built the data is not there to trigger the dag construction, the dag won't be in the bundle at all)
It really is when the dag is defined after the module ends that causes trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143959886
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
Changed "Attributed" for "Resolved / attached" source file. Which reads better indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143890826
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
External conditions are fine, as long as the condition is met a 'built' time.
That's also a requirement for having the dag in the output bundle (condition for being able to run this dag).
If the dag isn't definable at built time, cause for instance it depends on some external data not fulfill at build time I believe it won't run at all in the current artifact + manifest setup. It's another use case, not supported by the ts sdk I believe atm.
It really is when the dag is defined after the module ends that causes trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143890826
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
External conditions are fine, as long as the condition is met a 'built' time.
That's also a requirement for having the dag at all in the output bundle (condition for running it).
It really is when the dag is defined after the module ends that causes trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143890826
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
External conditions are fine, as long as the condition is met a 'built' time.
That's also a requirement for having the dag in the output bundle (condition for being able to run this dag).
If the dag isn't definable at built time, cause for instance it depends on some external data not fulfill at build time or we do not rebuild the project at all. I believe it won't run at all in the current artifact + manifest setup. It's another use case, not supported by the ts sdk I believe atm so I didn't address this as part of the 'dag source' problem solved by this ADR.
It really is when the dag is defined after the module ends that causes trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #73948:
URL: https://github.com/apache/airflow/pull/73948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add ADR for per-Dag source in the Lang-SDK bundle artifact [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #73948:
URL: https://github.com/apache/airflow/pull/73948#discussion_r4143890826
##########
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md:
##########
@@ -0,0 +1,129 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0015: Per-Dag Source in the Lang-SDK Bundle Artifact
+
+## Status
+
+Proposed
+
+> **Note:** The TypeScript SDK is the reference implementation, merged in
+> [apache/airflow#73723](https://github.com/apache/airflow/pull/73723). Go and Java adoption, and
+> the Airflow-core plumbing that surfaces this source in the UI Code tab, are follow-up work.
+
+## Context
+
+A native Lang-SDK Dag is authored entirely in the SDK's language, with no Python `@task.stub` file
+behind it ([ADR-0010](0010-native-dag-processing.md)). Its source is therefore the only source there
+is to show in the Code tab.
+
+[ADR-0003](0003-pure-java-dags.md) packed one `.java` file into the JAR, named by the
+`Airflow-Java-SDK-Dag-Code` manifest attribute — a single source per bundle. That is enough only
+while a bundle defines one Dag in one file. A real native bundle defines several Dags across several
+files: one entry module imports a module that constructs a Dag, and a single embedded source then
+shows the entry file for a Dag whose `new Dag(...)` lives in an imported module. Every Dag would show
+the same, mostly-unrelated file.
+
+[ADR-0006](0006-no-lang-sdk-source-display.md) settled the neighbouring case — a **mixed-language**
+Dag (Python owns the Dag, the Lang SDK only supplies task handlers) shows only its Python file — and
+deferred the native case to "the normal single-file path". [ADR-0010](0010-native-dag-processing.md)
+then flagged the gap directly: `get_source_code` is abstract, "a native Lang-SDK Dag has no Python
+source to return. What it should return, and how that squares with ADR-0006, is not settled here."
+This ADR settles it: what a native Lang-SDK bundle stores for display, and what a reader returns per
+Dag.
+
+## Decision
+
+### 1. Store each Dag-defining file, attributed per Dag, best effort
+
+The packer records, per native Dag, the source file the Dag was declared in, and embeds that file
+verbatim. The mapping is metadata (`dag_source_paths: {dag_id: path}` in the TypeScript bundle); the
+files are embedded regions alongside the compiled artifact.
+
+Attribution is **best effort**, and it covers the shapes authors actually use. A `new Dag(...)` at a
+module's top level — directly, in a `for` loop, or in a factory the module calls while it evaluates
+(before or after an `await`) — is recorded against the file whose module body was running when it was
+constructed. Dynamically generated Dags are therefore attributed like any other and are fully
+supported. The one shape it cannot attribute is a Dag constructed *after* its module has finished
+evaluating — from a detached callback such as `setTimeout` or a floating `.then` — which has no entry
+and is handled by the fallback below rather than by guessing. That pattern is non-idiomatic for a Dag
+file.
+
+### 2. De-duplicate by path
+
+Source content is stored **once per unique file path**. Several Dags declared in one file all map to
+that one path, and the file's bytes appear once in the artifact, never once per Dag. The per-Dag map
+is `dag_id → path`; the stored content is keyed by `path`. This is the property every SDK must
+preserve: a bundle with twenty Dags in two files embeds two files, not twenty.
+
+### 3. Always embed the entrypoint, as the fallback
+
+The entry file passed to the packer is **always** embedded and named (`entrypoint_path` in the
+TypeScript metadata), whether or not it declares a Dag. It is the fallback source: a reader asked for
+a native Dag that has no attributed file of its own returns the entrypoint rather than nothing, so a
+Dag the packer could not tie to a file still shows the file that assembles the bundle. This restores
+the guarantee the single-source format ([ADR-0003](0003-pure-java-dags.md)) gave — there is always *a*
+source — without giving up per-Dag attribution for the Dags that have it (which is the common case,
+generated Dags included).
+
+### 4. Mixed-language Dags carry no Lang-SDK source
+
+A Dag the bundle only supplies task handlers for is owned by Python, which owns its Code tab
+([ADR-0006](0006-no-lang-sdk-source-display.md)). The bundle stores no source for it, and a reader
+returns nothing — never the entrypoint — so the caller shows the Python file. The reader distinguishes
+the two by whether the caller asks for a specific Dag's source at all: a native Dag is looked up (and
+falls back to the entrypoint), a Python-owned Dag is not looked up and yields nothing.
+
+### 5. Every Lang SDK follows this
+
+TypeScript is the reference ([#73723](https://github.com/apache/airflow/pull/73723)). The Go and Java
+packers adopt the same shape — per-Dag source attribution, de-duplication by path, and an
+always-embedded entrypoint fallback — so a bundle reader treats every language the same way and the
+Code tab behaves identically regardless of which SDK produced the artifact.
+
+## What a reader returns, per Dag
+
+| the Dag is… | reader returns |
+|---|---|
+| native, with an attributed file (the common case — includes loop/factory-generated Dags) | that file's source |
+| native, but the packer could not attribute a file (a detached-callback construction) | the entrypoint source (fallback) |
+| mixed-language (Python-owned) | nothing — Python owns the Code tab |
Review Comment:
External conditions are fine, as long as the condition is met a 'built' time.
That's also a requirement for having the dag in the output bundle (condition for being able to run this dag).
If the dag isn't definable at built time, cause for instance it depends on some external data not fulfill at build time I believe it won't run at all in the current artifact + manifest setup.
It really is when the dag is defined after the module ends that causes trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org