You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/07/07 13:08:02 UTC
[PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
jason810496 opened a new pull request, #69545:
URL: https://github.com/apache/airflow/pull/69545
# Why
Release/backport branches (`v3-1-test`, `v3-2-test`, ...) often lack `go-sdk`/`java-sdk` entirely or
carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would
silently run the lang-SDK coordinator k8s test against missing or outdated SDK sources.
## What
- Added `_lang_sdk_fetch_upstream_sdk_sources()`: shallow-fetches upstream `main` (`upstream` remote if
configured, else the canonical GitHub URL) and extracts `go-sdk`/`java-sdk` into a throwaway staging
dir via `git archive`, never touching the real working tree.
- Go build now runs against a scratch workspace mirroring `go_example`'s relative layout with `go-sdk`
swapped for the upstream copy (its `go.mod` `replace` is a filesystem path, so this avoids editing
it). Java build simply points gradle at the upstream `java-sdk` copy instead of the local one.
- `airflow-core/`, `task-sdk/`, and the test's own harness fixtures (`go_example`/`java_example`) are
untouched and keep tracking the checked-out branch. `providers/` is out of scope (unused by this test;
overlaying it into the shared PROD image build isn't worth the invasiveness).
- Lives entirely in breeze (`kubernetes_commands.py`), so it applies to both CI and local
`breeze k8s setup-lang-sdk-test` — no CI workflow changes needed.
Known tradeoff: this makes the lang-SDK test provisioning require network access to GitHub, which it
didn't before.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes, with help of Claude Code Sonnet 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #69545:
URL: https://github.com/apache/airflow/pull/69545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69545:
URL: https://github.com/apache/airflow/pull/69545#discussion_r3567831802
##########
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py:
##########
@@ -2507,37 +2508,120 @@ def deploy_cluster(
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
+# The Go/Java SDKs are always built from upstream main so branches with stale or missing
+# go-sdk/java-sdk copies still test current SDK sources. See kubernetes-tests/lang_sdk/README.md.
+LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git"
+LANG_SDK_UPSTREAM_REF = "main"
-def _lang_sdk_build_go_bundle(staging: Path, output: Output | None, *, native: bool = False) -> None:
+def _lang_sdk_fetch_upstream_sdk_sources(staging: Path, output: Output | None) -> tuple[Path, Path]:
+ """Extract go-sdk/ and java-sdk/ from upstream main into a throwaway staging dir.
+
+ Prefers the ``upstream`` remote when configured, falling back to the canonical GitHub URL
+ (CI has no ``upstream`` and ``origin`` may be a fork). Shallow fetch + ``git archive`` never
+ touch the working tree or index.
+
+ The real, local task-sdk is symlinked alongside the extraction because java-sdk's
+ ``sdk/build.gradle.kts`` reads a sibling ``../task-sdk/.../schema.json``. The gradle wrapper
+ scripts and jar are ``export-ignore`` (ASF LEGAL-570) so ``git archive`` drops them;
+ ``git show`` restores them, re-marking ``gradlew`` executable.
+ """
+ remotes = run_command(
+ ["git", "remote"], cwd=AIRFLOW_ROOT_PATH, output=output, capture_output=True, text=True, check=True
+ ).stdout.split()
+ fetch_source = "upstream" if "upstream" in remotes else LANG_SDK_UPSTREAM_GIT_URL
+ get_console(output=output).print(
+ f"[info]Fetching {LANG_SDK_UPSTREAM_REF} from {fetch_source} for the lang-SDK Go/Java sources"
+ )
+ run_command(
+ ["git", "fetch", "--depth=1", fetch_source, LANG_SDK_UPSTREAM_REF],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ sha = run_command(
+ ["git", "rev-parse", "FETCH_HEAD"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ text=True,
+ check=True,
+ ).stdout.strip()
+ get_console(output=output).print(f"[info]lang-SDK Go/Java sources pinned to upstream main @ {sha}")
+ extracted = staging / "upstream_lang_sdk_sources"
+ extracted.mkdir(parents=True, exist_ok=True)
+ archive_path = staging / "upstream_lang_sdk_sources.tar"
+ run_command(
+ ["git", "archive", "--format=tar", f"--output={archive_path}", sha, "--", "go-sdk", "java-sdk"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ run_command(["tar", "-xf", str(archive_path), "-C", str(extracted)], output=output, check=True)
+ for rel_path, mode in (
+ ("java-sdk/gradlew", 0o755),
+ ("java-sdk/gradlew.bat", 0o644),
+ ("java-sdk/gradle/wrapper/gradle-wrapper.jar", 0o644),
+ ):
+ restored = extracted / rel_path
+ restored.parent.mkdir(parents=True, exist_ok=True)
+ restored.write_bytes(
+ run_command(
+ ["git", "show", f"{sha}:{rel_path}"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ check=True,
+ ).stdout
+ )
Review Comment:
Fixed in 4526464e09
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #69545:
URL: https://github.com/apache/airflow/pull/69545#discussion_r3567831802
##########
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py:
##########
@@ -2507,37 +2508,120 @@ def deploy_cluster(
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
+# The Go/Java SDKs are always built from upstream main so branches with stale or missing
+# go-sdk/java-sdk copies still test current SDK sources. See kubernetes-tests/lang_sdk/README.md.
+LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git"
+LANG_SDK_UPSTREAM_REF = "main"
-def _lang_sdk_build_go_bundle(staging: Path, output: Output | None, *, native: bool = False) -> None:
+def _lang_sdk_fetch_upstream_sdk_sources(staging: Path, output: Output | None) -> tuple[Path, Path]:
+ """Extract go-sdk/ and java-sdk/ from upstream main into a throwaway staging dir.
+
+ Prefers the ``upstream`` remote when configured, falling back to the canonical GitHub URL
+ (CI has no ``upstream`` and ``origin`` may be a fork). Shallow fetch + ``git archive`` never
+ touch the working tree or index.
+
+ The real, local task-sdk is symlinked alongside the extraction because java-sdk's
+ ``sdk/build.gradle.kts`` reads a sibling ``../task-sdk/.../schema.json``. The gradle wrapper
+ scripts and jar are ``export-ignore`` (ASF LEGAL-570) so ``git archive`` drops them;
+ ``git show`` restores them, re-marking ``gradlew`` executable.
+ """
+ remotes = run_command(
+ ["git", "remote"], cwd=AIRFLOW_ROOT_PATH, output=output, capture_output=True, text=True, check=True
+ ).stdout.split()
+ fetch_source = "upstream" if "upstream" in remotes else LANG_SDK_UPSTREAM_GIT_URL
+ get_console(output=output).print(
+ f"[info]Fetching {LANG_SDK_UPSTREAM_REF} from {fetch_source} for the lang-SDK Go/Java sources"
+ )
+ run_command(
+ ["git", "fetch", "--depth=1", fetch_source, LANG_SDK_UPSTREAM_REF],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ sha = run_command(
+ ["git", "rev-parse", "FETCH_HEAD"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ text=True,
+ check=True,
+ ).stdout.strip()
+ get_console(output=output).print(f"[info]lang-SDK Go/Java sources pinned to upstream main @ {sha}")
+ extracted = staging / "upstream_lang_sdk_sources"
+ extracted.mkdir(parents=True, exist_ok=True)
+ archive_path = staging / "upstream_lang_sdk_sources.tar"
+ run_command(
+ ["git", "archive", "--format=tar", f"--output={archive_path}", sha, "--", "go-sdk", "java-sdk"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ run_command(["tar", "-xf", str(archive_path), "-C", str(extracted)], output=output, check=True)
+ for rel_path, mode in (
+ ("java-sdk/gradlew", 0o755),
+ ("java-sdk/gradlew.bat", 0o644),
+ ("java-sdk/gradle/wrapper/gradle-wrapper.jar", 0o644),
+ ):
+ restored = extracted / rel_path
+ restored.parent.mkdir(parents=True, exist_ok=True)
+ restored.write_bytes(
+ run_command(
+ ["git", "show", f"{sha}:{rel_path}"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ check=True,
+ ).stdout
+ )
Review Comment:
Fixed in 4526464e09 — applied the suggested `or b""`, and also guarded the remaining dry-run crash sites downstream (the Go bundle `copytree`/`copy` of never-built artifacts, the Java jar glob, and the artifact-upload jar lookup), so `breeze k8s setup-lang-sdk-test --dry-run` now completes end to end and prints the full provisioning command sequence. Added dry-run regression tests covering each site.
---
Drafted-by: Claude Code (Fable 5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #69545:
URL: https://github.com/apache/airflow/pull/69545#discussion_r3559391305
##########
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py:
##########
@@ -2507,37 +2508,120 @@ def deploy_cluster(
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
+# The Go/Java SDKs are always built from upstream main so branches with stale or missing
+# go-sdk/java-sdk copies still test current SDK sources. See kubernetes-tests/lang_sdk/README.md.
+LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git"
+LANG_SDK_UPSTREAM_REF = "main"
-def _lang_sdk_build_go_bundle(staging: Path, output: Output | None, *, native: bool = False) -> None:
+def _lang_sdk_fetch_upstream_sdk_sources(staging: Path, output: Output | None) -> tuple[Path, Path]:
+ """Extract go-sdk/ and java-sdk/ from upstream main into a throwaway staging dir.
+
+ Prefers the ``upstream`` remote when configured, falling back to the canonical GitHub URL
+ (CI has no ``upstream`` and ``origin`` may be a fork). Shallow fetch + ``git archive`` never
+ touch the working tree or index.
+
+ The real, local task-sdk is symlinked alongside the extraction because java-sdk's
+ ``sdk/build.gradle.kts`` reads a sibling ``../task-sdk/.../schema.json``. The gradle wrapper
+ scripts and jar are ``export-ignore`` (ASF LEGAL-570) so ``git archive`` drops them;
+ ``git show`` restores them, re-marking ``gradlew`` executable.
+ """
+ remotes = run_command(
+ ["git", "remote"], cwd=AIRFLOW_ROOT_PATH, output=output, capture_output=True, text=True, check=True
+ ).stdout.split()
+ fetch_source = "upstream" if "upstream" in remotes else LANG_SDK_UPSTREAM_GIT_URL
+ get_console(output=output).print(
+ f"[info]Fetching {LANG_SDK_UPSTREAM_REF} from {fetch_source} for the lang-SDK Go/Java sources"
+ )
+ run_command(
+ ["git", "fetch", "--depth=1", fetch_source, LANG_SDK_UPSTREAM_REF],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ sha = run_command(
+ ["git", "rev-parse", "FETCH_HEAD"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ text=True,
+ check=True,
+ ).stdout.strip()
+ get_console(output=output).print(f"[info]lang-SDK Go/Java sources pinned to upstream main @ {sha}")
+ extracted = staging / "upstream_lang_sdk_sources"
+ extracted.mkdir(parents=True, exist_ok=True)
+ archive_path = staging / "upstream_lang_sdk_sources.tar"
+ run_command(
+ ["git", "archive", "--format=tar", f"--output={archive_path}", sha, "--", "go-sdk", "java-sdk"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ run_command(["tar", "-xf", str(archive_path), "-C", str(extracted)], output=output, check=True)
+ for rel_path, mode in (
+ ("java-sdk/gradlew", 0o755),
+ ("java-sdk/gradlew.bat", 0o644),
+ ("java-sdk/gradle/wrapper/gradle-wrapper.jar", 0o644),
+ ):
+ restored = extracted / rel_path
+ restored.parent.mkdir(parents=True, exist_ok=True)
+ restored.write_bytes(
+ run_command(
+ ["git", "show", f"{sha}:{rel_path}"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ check=True,
+ ).stdout
+ )
Review Comment:
`breeze k8s setup-lang-sdk-test --dry-run` crashes here. In dry-run, run_command returns stdout="" (a str),so `write_bytes("")` raises `TypeError: a bytes-like object is required, not 'str'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #69545:
URL: https://github.com/apache/airflow/pull/69545#discussion_r3559382370
##########
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py:
##########
@@ -2507,37 +2508,120 @@ def deploy_cluster(
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
+# The Go/Java SDKs are always built from upstream main so branches with stale or missing
+# go-sdk/java-sdk copies still test current SDK sources. See kubernetes-tests/lang_sdk/README.md.
+LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git"
+LANG_SDK_UPSTREAM_REF = "main"
-def _lang_sdk_build_go_bundle(staging: Path, output: Output | None, *, native: bool = False) -> None:
+def _lang_sdk_fetch_upstream_sdk_sources(staging: Path, output: Output | None) -> tuple[Path, Path]:
+ """Extract go-sdk/ and java-sdk/ from upstream main into a throwaway staging dir.
+
+ Prefers the ``upstream`` remote when configured, falling back to the canonical GitHub URL
+ (CI has no ``upstream`` and ``origin`` may be a fork). Shallow fetch + ``git archive`` never
+ touch the working tree or index.
+
+ The real, local task-sdk is symlinked alongside the extraction because java-sdk's
+ ``sdk/build.gradle.kts`` reads a sibling ``../task-sdk/.../schema.json``. The gradle wrapper
+ scripts and jar are ``export-ignore`` (ASF LEGAL-570) so ``git archive`` drops them;
+ ``git show`` restores them, re-marking ``gradlew`` executable.
+ """
+ remotes = run_command(
+ ["git", "remote"], cwd=AIRFLOW_ROOT_PATH, output=output, capture_output=True, text=True, check=True
+ ).stdout.split()
+ fetch_source = "upstream" if "upstream" in remotes else LANG_SDK_UPSTREAM_GIT_URL
+ get_console(output=output).print(
+ f"[info]Fetching {LANG_SDK_UPSTREAM_REF} from {fetch_source} for the lang-SDK Go/Java sources"
+ )
+ run_command(
+ ["git", "fetch", "--depth=1", fetch_source, LANG_SDK_UPSTREAM_REF],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ sha = run_command(
+ ["git", "rev-parse", "FETCH_HEAD"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ text=True,
+ check=True,
+ ).stdout.strip()
+ get_console(output=output).print(f"[info]lang-SDK Go/Java sources pinned to upstream main @ {sha}")
+ extracted = staging / "upstream_lang_sdk_sources"
+ extracted.mkdir(parents=True, exist_ok=True)
+ archive_path = staging / "upstream_lang_sdk_sources.tar"
+ run_command(
+ ["git", "archive", "--format=tar", f"--output={archive_path}", sha, "--", "go-sdk", "java-sdk"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ check=True,
+ )
+ run_command(["tar", "-xf", str(archive_path), "-C", str(extracted)], output=output, check=True)
+ for rel_path, mode in (
+ ("java-sdk/gradlew", 0o755),
+ ("java-sdk/gradlew.bat", 0o644),
+ ("java-sdk/gradle/wrapper/gradle-wrapper.jar", 0o644),
+ ):
+ restored = extracted / rel_path
+ restored.parent.mkdir(parents=True, exist_ok=True)
+ restored.write_bytes(
+ run_command(
+ ["git", "show", f"{sha}:{rel_path}"],
+ cwd=AIRFLOW_ROOT_PATH,
+ output=output,
+ capture_output=True,
+ check=True,
+ ).stdout
+ )
Review Comment:
```suggestion
restored.write_bytes(
run_command(
["git", "show", f"{sha}:{rel_path}"],
cwd=AIRFLOW_ROOT_PATH,
output=output,
capture_output=True,
check=True,
).stdout
or b""
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin lang-SDK k8s test's Go/Java SDKs to upstream main [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69545:
URL: https://github.com/apache/airflow/pull/69545#issuecomment-4976347067
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/69895"><img src="https://img.shields.io/badge/PR-69895-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org