You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by GitBox <gi...@apache.org> on 2022/09/13 18:38:34 UTC
[GitHub] [buildstream] ssssam opened a new issue, #1751: Recc inside BuildStream
ssssam opened a new issue, #1751:
URL: https://github.com/apache/buildstream/issues/1751
BuildStream is a tool to execute full "package" builds, including awkward parts like `configure` scripts, stripping, integration commands, in a repeatable way using a sandbox. It can distribute these builds to build farm workers via the REAPI, at a granularity of one element => one worker. This means, that a large element like WebKit is only compiled on a single worker which can be slow.
[Recc](https://gitlab.com/BuildGrid/recc) is a tool to distribute individual C/C++ compile tasks across build farm workers. At present, it cannot be used inside a BuildStream sandbox because it has no way to access the REAPI endpoint, but ... could we allow a controlled opening in the sandbox to enable Recc to be used inside element builds?
The ideal scenario would be a large element building as follows:
1. `bst build` runs the element `configure-commands` on a single worker, with `recc` replacing `cc` etc.
2. `bst build` runs the element `build-commands`:
a. For each C/C++ compile operation, `recc` hashes the inputs for the given compile operation to produce a cache key, and checks the remote cache via configured REAPI endpoint
b. If needed, compile is invoked on a remote worker. Then, the object file is downloaded from the cache.
3. Link operations run on the main worker and still probably take ages to complete.
4. Profit!
This is not guaranteed to be faster than building on a single node, a lot depends on the infrastructure being used and the element being built. I think it would be worth exploring as part of a larger project to try and reduce build times of large BuildStream projects like gnome-build-meta.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on issue #1751: Recc inside BuildStream
Posted by GitBox <gi...@apache.org>.
gtristan commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1276970193
I've uploaded https://gitlab.com/tristanvb/buildstream-bazel-demo which proves our concept of having recc access the cas server from within the sandbox (using the #1772 branch), at least when running buildstream locally.
Will do some cleanup, possibly some performance testing, and then blog on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Recc inside BuildStream [buildstream]
Posted by "nanonyme (via GitHub)" <gi...@apache.org>.
nanonyme commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1868368432
@gtristan can you please submit proposal for actual integration that allows recc to run? freedesktop-sdk 24.08 and later will provide recc element.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on issue #1751: Recc inside BuildStream
Posted by GitBox <gi...@apache.org>.
gtristan commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1278582366
And a summary of our experiment can be found on my blog: https://blogs.gnome.org/tvb/2022/10/14/buildstream-at-apachecon-2022-new-orleans/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] staehle commented on issue #1751: Recc inside BuildStream
Posted by GitBox <gi...@apache.org>.
staehle commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1262623753
Interested in this, or at least something like it that can do caching and distributed builds within an element.
For context, I am evaluating BuildStream as a meta build system to migrate an embedded Linux OS for a large company, and we currently have a very large codebase for our core application that runs on our devices. The core application is not componentized like other items that build alongside it, such as Busybox or OpenSSL or ffmpeg that only take a minute or so to build from scratch. While the correct solution is to split up that large primary application into their own elements, some portions of it wouldn't be possible to do so, and for the other it would be at least challenging and time-consuming when you have a lot of developers within different departments/groups to coordinate with (bureaucracy, am I right?). When actively developing code within that codebase, BuildStream's current (very excellent otherwise) element-level caching and distribution doesn't help much, so a more fine-grained inner-element caching and distributed build solution would be needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] juergbi commented on issue #1751: Recc inside BuildStream
Posted by GitBox <gi...@apache.org>.
juergbi commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1245872553
This could certainly be interesting if either no big workers are available in a build farm or if the project could benefit from more fine-grained caching. The idea of using recc with nested remote execution has been around for a while but nobody has attempted to implement it yet, as far as I know.
I think this would mainly be a feature of the REAPI server/worker side. E.g. BuildGrid/BuildBox could support a platform property that allows network access to the CAS and Execution services and sets environment variables to communicate the URIs. It might make sense to extend buildbox-casd to work as an Execution service proxy, making it possible to restrict the sandbox hole to the worker's local buildbox-casd.
BuildStream would simply need to support setting that platform property (possibly in a generic way, see #1313). BuildStream projects could then include recc in their sandbox and configure it using the environment variables set by the worker.
If recc supported buildbox-casd's `CaptureFiles` method, it could skip hashing of input files when buildbox-fuse is used as the hash would already available in the `user.checksum.sha256` xattr, reducing the recc overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on issue #1751: Recc inside BuildStream
Posted by GitBox <gi...@apache.org>.
gtristan commented on issue #1751:
URL: https://github.com/apache/buildstream/issues/1751#issuecomment-1275784530
Interesting that you opened this issue and I didn't notice.
At ApacheCon @sstriker and I had a bit of a hackfest and came up with #1772 in order to test this out by building a bazel project with remote caching enabled, I will push that sample project up shortly, however it didn't work so far as I was not able to properly configure bazel to use remote caching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org