You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/07/05 10:33:10 UTC
[PR] Speed up lang-SDK k8s test by building Go/Java natively in CI [airflow]
potiuk opened a new pull request, #69411:
URL: https://github.com/apache/airflow/pull/69411
The Multi-Lang KubernetesExecutor system test (#68709) builds a Go bundle and a Java jar inside throwaway toolchain containers. On a dev host the Go module/build cache, the Gradle distribution and the dependency caches persist between runs, but CI runners are ephemeral and nothing cached them — so every scheduled run re-pulled the `golang`/`eclipse-temurin` images and re-downloaded the Gradle distribution and all dependencies. That added ~10 min to each of the six `KubernetesExecutor` + standard-naming-off variants (the only ones that run the test), which the daily CI duration-trend alert flagged.
This builds the artifacts with the **host toolchain** in CI instead, provisioned and cached through `actions/setup-go` and `actions/setup-java` (the same pattern the sibling *Go SDK tests* / *Java SDK tests* jobs already use). A new `LANG_SDK_NATIVE_TOOLCHAIN=true` env, set only for that variant, makes breeze invoke the host `go` / `./gradlew` rather than `docker run`. Local runs keep the containerised build, so a dev host still needs neither Go nor a JDK installed.
Measured on run `28635348211`: the parallel provisioning block was ~5m36s, almost entirely toolchain-image pulls + cold Gradle/Maven/Go dependency downloads. Removing the image pulls and warming the caches should cut that to ~1–1.5 min (~4 min/job); the subsequent helm upgrade and the test itself are unaffected.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up lang-SDK k8s test by building Go/Java natively in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #69411:
URL: https://github.com/apache/airflow/pull/69411#issuecomment-4885764725
> I like the idea of having "native" host setup for CI to speed up the process! Thanks a lot.
Yeah - while not **entirely** reproducible, we have the docker run that a) caches nicely for local use b) is reproducible, so we are good with native builds on CI - cache on GitHub is **very fast**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up lang-SDK k8s test by building Go/Java natively in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #69411:
URL: https://github.com/apache/airflow/pull/69411#issuecomment-4885908967
Looks good - Back to 13-15 m from 25!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up lang-SDK k8s test by building Go/Java natively in CI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #69411:
URL: https://github.com/apache/airflow/pull/69411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Speed up lang-SDK k8s test by building Go/Java natively in CI [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69411:
URL: https://github.com/apache/airflow/pull/69411#issuecomment-4885914350
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/28739609847'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/673183b2c8128095f07c3b5811cc12c24707b3e7"><img src='https://img.shields.io/badge/Commit-673183b-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 673183b v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org