You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by "lupyuen (via GitHub)" <gi...@apache.org> on 2026/03/22 04:03:41 UTC
[PR] [Do Not Merge] Testing of CI Build Retry [nuttx]
lupyuen opened a new pull request, #18576:
URL: https://github.com/apache/nuttx/pull/18576
__Please don't merge this PR!__ This is a test of the updated Build Script for Retrying CI Builds upon failure (e.g. dependency download failed). We're testing in the Main NuttX Repo, because our own repos won't reproduce the Dependency Download Failure. More details:
- [Apache NuttX RTOS: Analysing the Most Expensive PR](https://youtu.be/swFaxaTCEQg)
- [Apache NuttX RTOS: Retrying a CI Build](https://www.youtube.com/watch?v=LBhEJ6Qf4iM)
- https://github.com/apache/nuttx/issues/18562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "lupyuen (via GitHub)" <gi...@apache.org>.
lupyuen commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4252202953
@hartmannathan Awesome ideas! I was pondering your ideas with @simbit18, we hit a couple of roadblocks:
1. How will we auto-refresh the Cached Dependencies in nuttx-ci-deps? To make sure they are always up-to-date?
2. ESP32 Downloads might be tricky, I don't see any obvious way to hook the ESP32 Builds to our Download Script. Also: I think ESP32 Downloads might need to be refreshed more often.
3. Nearly all Download Failures are for URLs hosted at github.com. So we don't have much problems with downloads from External Third Parties. I suspect github.com is blocking our downloading because we run too many CI Jobs in parallel, all spamming github.com at the same time. (Hence I implemented Random Exponential Backoff)
4. FYI chromium.googlesource.com was blocking our download of libyuv library, so we cached it at the NuttX Mirror Repo. libyuv hasn't been updated since 2021, so we probably won't need to worry about refreshing:
- https://github.com/apache/nuttx-apps/pull/3449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "lupyuen (via GitHub)" <gi...@apache.org>.
lupyuen commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4257325622
Thanks @hartmannathan! Sorry for messing up this complex discussion, maybe we should move it to the [Project TODOs](https://github.com/orgs/apache/projects/455/views/2). I think to roll out any meaningful updates to NuttX CI (like Build Dependency Caching) we need to...
1. Identify the __NuttX CI Lead__ who will implement, monitor and maintain these CI Enhancements
2. Sorry I can't be CI Lead: I suffer from hypertension and I can't commit to long-term projects. I believe @simbit18 might be busy with commitments (we should ask). So we might need to hunt for a suitable CI Lead.
3. To assist the CI Lead: We should nominate __Arch Owners__ (i.e. Owner for everything Arm32 / Arm64 / RISC-V / Xtensa / ...) and __Board Representatives__ (i.e. Representative for STM32 Boards / RP2040 Boards / IMX Boards / ...)
4. So it's easier for NuttX CI Lead to consult the Arch Owner on any CI Design Decisions. And Arch Owner can talk to the Board Reps, on behalf of CI Lead, [as explained here](https://youtu.be/JEF9_BqHWaY?si=lBM6RsMkw3agt7y5&t=3095). Can't find the ESP32 Downloads? We ask the Xtensa Owner :-)
5. As for __Non-ESP32 Downloads:__ NuttX has [73 curl commands](https://github.com/search?q=repo%3Aapache%2Fnuttx-apps+curl+language%3AMakefile&type=code&p=2&l=Makefile) for downloading Build Dependencies. We could rename the 73 `curl` commands to `nuttx-curl.sh`, which will read from our Dependency Cache. And ask Arch Owners + Board Reps to help test all 73 `nuttx-curl.sh` commands.
6. Remember: `nuttx-curl.sh` needs to support all kinds of Local PCs for development: macOS, WSL, Ubuntu, FreeBSD, etc. Maybe on Local PCs: We simply point `nuttx-curl.sh` to `curl`?
7. Will we have an __Operations Person__ (paid, preferably) to monitor NuttX CI daily? Like [checking the Daily Builds](https://github.com/apache/nuttx/issues/18659)? Making sure that Build Dependencies are cached correctly and updated often (e.g. due to security patches)? This person will affect our Upcoming CI Redesign. That's why I implemented CI Build Retry in the simplest way, that requires Zero Maintenance, Zero Monitoring.
<img width="1910" height="1071" alt="vlcsnap-2026-03-09-09h04m27s740" src="https://github.com/user-attachments/assets/1a3565eb-997b-4ae8-a45b-5c6ff1c5d479" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "simbit18 (via GitHub)" <gi...@apache.org>.
simbit18 commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4251039294
@lupyuen Thank you so much !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "simbit18 (via GitHub)" <gi...@apache.org>.
simbit18 merged PR #18576:
URL: https://github.com/apache/nuttx/pull/18576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "lupyuen (via GitHub)" <gi...@apache.org>.
lupyuen commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4251368387
Thanks @hartmannathan for the Minimum Backoff suggestion! I'll monitor the builds over the next few days to watch for any Download Errors, whether we need to set a Minimum Backoff. Our situation is a little delicate right now: Imposing a Minimum Backoff might mean that Compile Errors will take longer to complete the retries, holding up our devs. I'll also watch out for CI Test Errors, which will have longer retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "hartmannathan (via GitHub)" <gi...@apache.org>.
hartmannathan commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4254405879
> @hartmannathan Awesome ideas! I was pondering your ideas with @simbit18, we hit a couple of roadblocks:
Thanks @lupyuen @simbit18 ! I will try to answer:
>
>
>
> 1. How will we auto-refresh the Cached Dependencies in nuttx-ci-deps? To make sure they are always up-to-date? Who will watch over nuttx-ci-deps, and fix it if any downloads fail?
>
Interesting, I hadn't considered auto-refresh. My thoughts were that the nuttx-ci-deps repo would be populated with specific versions of the packages, that we would consider "blessed" versions for CI testing. When a NuttX release happens, we could document which versions of dependencies have been used in testing. Developers who wish to use more bleeding edge versions would of course have the option to do that. Perhaps after each release, we could evaluate the versions of dependencies that are available and decide whether to update the nuttx-ci-deps repo. This means there is a human in the loop who can verify that package is legit and so on.
>
>
> 2. ESP32 Downloads might be tricky, I don't see any obvious way to hook the ESP32 Builds to our Download Script. Also: I think ESP32 Downloads might need to be refreshed more often.
I'm not familiar with ESP32 so I'm unsure how to answer this. Could you tell me a bit more about how it gets downloaded?
>
>
>
> 3. Nearly all Download Failures are for URLs hosted at github.com. So we don't have much problems with downloads from External Third Parties. I suspect github.com is blocking our downloading because we run too many CI Jobs in parallel, all spamming github.com at the same time. (Hence I implemented Random Exponential Backoff)
>
In this case, maybe there should be a short (less than 10 seconds maybe) random delay before each download starts? That might cause the requests to be more serialized and perhaps avoid or reduce the failures.
>
>
> 4. FYI chromium.googlesource.com was blocking our download of libyuv library, so we cached it at the NuttX Mirror Repo. libyuv hasn't been updated since 2021, so we probably won't need to worry about refreshing:
>
> - https://github.com/apache/nuttx-apps/pull/3449
This was exactly my thought, but for all downloaded deps. Our cached libyuv lib could move to a nuttx-ci-deps repo if we go this route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Do Not Merge] Testing of CI Build Retry [nuttx]
Posted by "lupyuen (via GitHub)" <gi...@apache.org>.
lupyuen commented on code in PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#discussion_r2986976383
##########
tools/testbuild.sh:
##########
@@ -580,6 +580,57 @@ function dotest {
fi
}
+# Build one entry from the test list file. Retry 5 times on failure.
+
+function retrytest {
+ # Remember the Fail Status and clear it for each build
+ local line=$1
+ local prevfail=$fail
+ local backoff=60 # Initial Exponential Backoff, in seconds
+
+ # Build and retry 5 times on failure, with Random Exponential Backoff
+ local maxbuilds=6
+ for ((i = 1; i <= $maxbuilds; i++)); do
+ echo "Build Attempt $i"
+ fail=0
+ dotest $line
+
+ # Don't retry if the build succeeded
+ if [ ${fail} -eq 0 ]; then
+ break
+ else
+ # Build Failed: Clean up any corrupted downloads, don't reuse
+ git clean -fd
+ git status
+ pushd ../apps
+ git clean -fd
+ git status
+ popd
+ fi
Review Comment:
Thanks! Fixing now :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Do Not Merge] Testing of CI Build Retry [nuttx]
Posted by "simbit18 (via GitHub)" <gi...@apache.org>.
simbit18 commented on code in PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#discussion_r2986966959
##########
tools/testbuild.sh:
##########
@@ -580,6 +580,57 @@ function dotest {
fi
}
+# Build one entry from the test list file. Retry 5 times on failure.
+
+function retrytest {
+ # Remember the Fail Status and clear it for each build
+ local line=$1
+ local prevfail=$fail
+ local backoff=60 # Initial Exponential Backoff, in seconds
+
+ # Build and retry 5 times on failure, with Random Exponential Backoff
+ local maxbuilds=6
+ for ((i = 1; i <= $maxbuilds; i++)); do
+ echo "Build Attempt $i"
+ fail=0
+ dotest $line
+
+ # Don't retry if the build succeeded
+ if [ ${fail} -eq 0 ]; then
+ break
+ else
+ # Build Failed: Clean up any corrupted downloads, don't reuse
+ git clean -fd
+ git status
+ pushd ../apps
+ git clean -fd
+ git status
+ popd
+ fi
Review Comment:
Hi @lupyuen it might be better to use
git -C $APPSDIR clean -fd
git status
git -C $nuttx clean -fd
git status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "lupyuen (via GitHub)" <gi...@apache.org>.
lupyuen commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4285731769
I made a video that explains the Lessons Learnt from this PR. Because one year from today: None of us will remember why we're doing all this :-)
- [Apache NuttX RTOS: CI Build Retry works OK yay!](https://youtu.be/dT8VppS4qR8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CI: Retry build upon failure [nuttx]
Posted by "hartmannathan (via GitHub)" <gi...@apache.org>.
hartmannathan commented on PR #18576:
URL: https://github.com/apache/nuttx/pull/18576#issuecomment-4252021089
@lupyuen Sure, we can try it in its current form for now.
Meanwhile I had an idea that might be worth exploring:
In this idea, we would create a special repository on GitHub; suggested name: nuttx-ci-deps, meaning NuttX CI Dependencies. In that repository, we would place a copy of every dependency we are currently downloading from third parties. This would be used for CI builds only, not by normal developers or users.
How would it work? Our GitHub CI scripts could pass a special command line argument to the NuttX build scripts (make or cmake).
In the NuttX build scripts, the special argument would cause the download logic to get files from nuttx-ci-deps instead of downloading from third parties.
The rationale behind this idea is that downloading from third parties introduces a failure mode that is separate from GitHub. It could be, for example, that some third parties limit downloads originating from GitHub for the same reasons as GMP: too many downloads. That might cause some of our download failures. Downloading from third parties again and again for every build is also unfair to them. Furthermore, I would assume that GitHub probably has a more efficient and reliable path to get data from itself than from third parties.
One more idea:
Putting the retry logic in the GitHub CI scripts has the (known) disadvantage of retrying builds that fail due to compiler errors. In those cases we want to stop the build. We might accomplish that in the following way:
A script could be implemented at tools/download.sh. This script would encapsulate all download-related logic for the NuttX build system. It would know how to run curl, wget, fetch, and git, to retrieve and validate files when called for by the config. This script could also encapsulate the retry logic. Finally, this script could check the special argument and override the URL to get the files from nuttx-ci-deps instead of the normal download location.
Thoughts?
Just to clarify, yes, we should try this PR as-is and see if it improves our GHA usage. If we aren't satisfied, the ideas above are additional avenues for exploration...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org