You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "weiqingy (via GitHub)" <gi...@apache.org> on 2026/07/10 06:48:58 UTC
[I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
weiqingy opened a new issue, #889:
URL: https://github.com/apache/flink-agents/issues/889
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
The `it-java` and `it-python` CI jobs fail intermittently for reasons unrelated to the code under test. A [recent run](https://github.com/apache/flink-agents/actions/runs/29032939015) shows both failure modes at once, with every unit-test job green.
**1. The Ollama install step fails on a transient network blip.** `tools/start_ollama_server.sh` pipes `https://ollama.com/install.sh` into `sh`. That upstream script probes for `ollama-linux-${ARCH}.tar.zst` with a `curl --fail --silent --head`, and on any failure of that single probe it falls through to a `.tgz` asset. Current releases publish only `.tar.zst` — `.tgz` returns `404`. So one flaky HEAD request turns into a hard failure:
```
>>> Downloading ollama-linux-amd64.tgz
curl: (22) The requested URL returned error: 404
gzip: stdin: unexpected end of file
tar: Child returned status 1
##[error]Process completed with exit code 2.
```
Two `it-java` jobs failed this way, before compiling or running a single test. Retrying the install would recover, since the underlying asset does exist.
**2. The wrapper masks a failed download.** In `tools/start_ollama_server.sh`, `ret=$?` is read after `curl -fsSL ... | sh`, so it captures the exit status of `sh`, not `curl`. Without `pipefail`, a failed `curl` leaves `sh` reading empty stdin and exiting `0` — CI then proceeds with no Ollama installed and fails later somewhere less obvious.
**3. Integration jobs have no runtime bound.** No job in `ci.yml` sets `timeout-minutes`. In the run above, `it-python [python-3.12] [flink-2.2]` hung and was killed at GitHub's 6-hour ceiling (`6h0m16s`), consuming shared runner capacity for a result that was knowable in minutes. A `timeout-minutes` on the integration jobs would surface the same failure far sooner.
**Open question — the hang itself.** In that job Ollama installed correctly, then `llama-server` crashed while serving `qwen3:1.7b`:
```
ollama._types.ResponseError: llama-server process has terminated: signal: segmentation fault (core dumped)
```
Every failing test there is Ollama-backed; the `py4j` errors are downstream of the dead model server. I don't yet have evidence for the root cause (upstream regression, runner memory, or model artifact). One option is pinning `OLLAMA_VERSION` — `install.sh` supports it, and the workflow already pins `uv` to `0.11.0` in the adjacent step — which would at least make the toolchain deterministic. I'd welcome opinions on whether that's worth doing, and on how the project wants to handle bumping such a pin.
If this direction sounds right, I'd propose splitting the work into two independent PRs: the retry plus `pipefail` fix in `tools/start_ollama_server.sh`, and the `timeout-minutes` addition in `ci.yml`.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #889:
URL: https://github.com/apache/flink-agents/issues/889#issuecomment-4932856612
Will work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #889:
URL: https://github.com/apache/flink-agents/issues/889#issuecomment-4939686165
Status map for anyone picking this up, since the work has spread across a few PRs and a sibling issue.
This issue produced two independent PRs:
- **#891** — retry the Ollama install and propagate download failures (fixes the transient `.tgz` 404).
- **#892** — bound the integration-test jobs with `timeout-minutes` so a hung job fails fast instead of running to the 6-hour ceiling.
Both currently show red CI, but the red is **not** these changes. The Ollama install succeeds in every failing run; only the Ollama-backed integration tests crash, and the failing jobs roam from run to run while the changes stay the same. That intermittent flake is tracked separately in **#893**.
Root cause (detail in #893): a subset of GitHub `ubuntu-latest` runner VMs trip an upstream `llama.cpp` segmentation fault while serving `qwen3:1.7b` on CPU. A given VM is effectively deterministic, so re-running the *whole* matrix keeps hitting some cursed VM. The dated onset lines up with a GitHub runner-image rollout — and notably the installed Ollama version has been constant across the green→red boundary, so pinning the Ollama version does not apply here.
The fix is **#895**: a `workflow_run` companion that re-runs only the failed integration-test legs on a *fresh* runner (which re-rolls the VM), capped at two attempts, with unit/lint/build failures never re-run. It was validated end-to-end on a fork. One caveat noted on that PR: `workflow_run` companions only run from the default branch, so #895 cannot heal its own CI and takes effect only once merged.
Suggested order, entirely at the maintainers' discretion: **#895 first** — once merged it heals the integration-test flake for every subsequent run, including #891 and #892 — then #891/#892, which are otherwise green apart from the flake. They can also be reviewed independently. I'll leave the sequencing and the decisions to the maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 closed issue #889: [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs
URL: https://github.com/apache/flink-agents/issues/889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #889:
URL: https://github.com/apache/flink-agents/issues/889#issuecomment-4939750318
@wenjin272 When you get a chance, could you take a look at this?
CI has been failing more often recently, so it would be good to mitigate this to make PR creation, validation, and review smoother. My suggestion is to prioritize #895 first, since the fresh-runner auto-rerun would unblock the flaky integration-test jobs and give us the most immediate impact.
The map above has the full context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on issue #889:
URL: https://github.com/apache/flink-agents/issues/889#issuecomment-4955470507
@weiqingy Regarding issue #895, I suggest we first evaluate whether we can resolve the llama server segmentation fault before deciding if launching a new runner for retries is necessary when CI fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Tech Debt] Harden Ollama install and bound runtime for integration-test CI jobs [flink-agents]
Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on issue #889:
URL: https://github.com/apache/flink-agents/issues/889#issuecomment-4966429791
Agreed, a real fix beats a retry, so let's hold #895 until we know whether the segfault can be resolved at the source.
I've put the details on #893. The hardware lead looks right, but `OLLAMA_LLM_LIBRARY` won't get us there on 0.31.x, since it only filters GPU library directories now. The CPU kernel is chosen by `llama-server`, which loads the sapphirerapids ggml variant, the only one built with AMX, on those hosts. So the fix would be to stop that variant loading, in `tools/start_ollama_server.sh`.
Next step I'd suggest is a small diagnostic in the integration jobs to log the CPU flags and the loaded variant, so we can confirm it rather than guess. If it holds, I'll put up the fix and close #895.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org