You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "bryancall (via GitHub)" <gi...@apache.org> on 2026/10/14 21:21:24 UTC
[I] Branch build clones: mirror stalls, clone timeout and controller memory [trafficserver]
bryancall opened a new issue, #13839:
URL: https://github.com/apache/trafficserver/issues/13839
Branch builds keep failing in the Clone stage, and the controller keeps running out of memory. I dug into it on Oct 13 and 14, and there are three separate problems:
1. **ATS reused connections to the mirror after the mirror had closed them.** Each reuse sat for 180 s before ATS retried. On Oct 13 there were 30 of these stalls on the mirror, 23 on the docs site and 8 on Jenkins.
2. **Branch builds fetch every branch at depth 1000.** One fetch is a 165 MB pack and about 570 MB of git memory on the controller. A branch build starts about 30 children at once on an 8 GB box, so it runs out of memory. There were 32 kernel out-of-memory kills from Sep 2 to Oct 13, and 4 more on Oct 14. Fetching only the branch being built is 37 MB and about 306 MB.
3. **The 1-minute clone timeout wrapped all three retries.** A single fetch takes about 15 s when the box is quiet, so the cap only failed under the load above. Because the retries were inside it, they never got a real chance.
Recent master branch builds:
| Build | Children passed |
|---|---|
| [#1423](https://ci.trafficserver.apache.org/job/master/job/branch_build/1423/) | 8 of 29 |
| [#1424](https://ci.trafficserver.apache.org/job/master/job/branch_build/1424/) | 6 of 29 |
### Done
- [x] Raise the branch clone timeout from 1 minute to 10 ([apache/trafficserver-ci#473](https://github.com/apache/trafficserver-ci/pull/473))
- [x] Stop ATS reusing connections to the mirror (`server_session_sharing.match=none` on the `/mirror/` remap, controller config)
- [x] Cap the mirror at 16 httpd workers and 2 GB of memory (controller config)
### To do
- [ ] Fetch only the branch being built in the branch pipelines (PR to come)
- [ ] Give each checkout retry its own timeout: `retry(3) { timeout(...) { checkout } }` (PR to come)
- [ ] Add a timeout and retry to the trafficserver-ci checkout in the branch pipelines
- [ ] Apply the connection fix to the docs and Jenkins remaps as well, and save it in the config so it survives a restart
- [ ] Turn off ATS debug logging on the controller (about 3.3 GB an hour of `traffic.out`)
- [ ] Shrink the ATS RAM cache on the controller (1 GB today)
- [ ] Check the next branch build's fan-out against the new 2 GB mirror limit. About 3 all-branch fetches fit in 2 GB, so git may get killed inside the container until fetches are branch-only.
- [ ] Find out why freebsd, osx and osx-m1 fail in Clone at about 180 s ("3775 bytes of body are still expected")
- [ ] Fix the mirror webhook leaving thousands of finished `git` processes behind
- [ ] Point the 10.1.x and 10.2.x branch pollers at their own branches (both watch 10.0.x)
- [ ] Free disk on the controller (90% full)
- [ ] File an ATS issue for reusing a backend connection the backend already closed
### Lower priority
- [ ] Repack the mirror into one pack (about 20% faster fetches)
- [ ] Remove the duplicate `connect_attempts_timeout` and rename the deprecated `connect_attempts_max_retries_down_server` in the controller's records.yaml
The pipeline PRs are in [apache/trafficserver-ci](https://github.com/apache/trafficserver-ci), which has issues turned off, so this lives here. PRs for these items should mention this issue so they link back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org