You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Adrian Vovk <ad...@gmail.com> on 2025/02/17 00:08:38 UTC
New type of plugin that generates multiple elements
Hi everyone,
I've got a proposal for some new functionality in BuildStream. Forgive
me for the long email... I'm told I need to email here before I try
writing any code. So, I'd like to first motivate the new feature, and
then explain what I think it should look like.
# Background
One of my strategic goals for carbonOS (and now GNOME OS) has been to
support device-specific images wherever appropriate. Device-specific
images are, by nature, all very similar to each other: just some extra
packages here, some partition table tweaks there, and so on. Build tools
that deal with this (for example: Android's[1][2], ChromeOS's[3][4], and
postmarketOS's[5][6]) have all converged on similar solutions. They all
have a unified and highly parameterized image build pipeline, and
device-specific configuration files that control the pipeline's numerous
settings.
Buildstream, as it currently is, isn't well suited to create this
optimal image-build pipeline. But why not? Well, composing a collection
of buildstream elements into a single OS image generally requires more
than one element definition: you need at least one `stack` to group up
the packages that'll go into your image, then you need a
`collect-manifest` to generate your SBOM, maybe a couple of `compose`
elements to build your initrd and root directory trees, and finally some
`script`s to shove everything into the final disk images. This is
perfectly fine if you're building one image, but if you want to build a
second device-specific variant you need to copy and then edit these
elements. This quickly becomes unmaintainable. There's no mechanism in
Buildstream to parameterize across element boundaries like we'd need to
here.
My solution for this in carbonOS was a workaround. I added a "code
generation" script [7], that parses a device-specific configuration file
[8] and generates the necessary per-device elements for me. The bash
script was definitely crude, but it did give me the solution I wanted:
For each device, I could write a config file along with some
device-specific package definitions, while reusing the behavior of a
unified image build pipeline.
# Proposed Solution
I would like to introduce a new type of plugin to BuildStream, which I'm
tentatively calling a "generator". Much like an element, each instance
of a generator plugin corresponds to a generator definition file
(.bstg). Much like an element, we decide which plugin to instantiate
from the `kind:` field found in a generator definition file. A
generator's job would be to transform the contents of its definition
file into a dict of `Node`s that can then be parsed into BuildStream
elements as if they were read from disk.
Each entry in the dict maps a filename to the root `Node` of the
corresponding generated element. This allows other elements, from
outside of the generator, to refer to elements created by the generator.
For example, let's say there's a generator defined at
elements/something/demo.bstg, which generates two elements: foo.bst and
bar.bst. Elements anywhere in the project can refer to these generated
elements via something/demo.bstg/foo.bst and something/demo.bstg/bar.bst
respectively. From the perspective of other elements, a generator
appears like a directory that contains normal element definitions inside.
I think the easiest way to explain how this generator system would work
in practice would be via examples. So let's do that. Here's what the
simplest generator plugin might look like (pseudocode):
```
# plugins/generators/basic.py
class BasicGenerator(Generator):
def __init__(self, def: Node):
self.elements = def["elements"]
def generate(self) -> Dict[str, Node]:
return self.elements
```
This generator just lets you define multiple elements in one file. It
does nothing special other than that. A corresponding definition file
might look like this:
```
# elements/demo.bstg
kind: basic
elements:
foo.bst:
kind: meson
[...]
foo-filtered.bst:
kind: filter
build-depends:
- demo.bstg/foo.bst
[...]
```
Of course, the point of these generator plugins would be to do something
more interesting. So, here's an example of what a generator implementing
our device-specific image build pipeline would look like:
```
# elements/boards/microsoft-surface/board.bstg
kind: device
description: Board definition for the Microsoft Surface series of devices
supported_arches:
- x86_64 # Most devices before 2024
- aarch64 # Devices after 2024 switched to Arm
kernel: boards/microsoft-surface/linux-surface.bst
extra_packages:
- boards/microsoft-surface/webcam-firmware.bst
- boards/microsoft-surface/extra-power-profiles.bst
- boards/microsoft-surface/detatchable-keyboard-daemon.bst
- [...]
append_cmdline: >-
intel.iommu=force
qcom.iommu=enable
[...]
```
```
# plugins/generators/device.py
class Device(Generator):
def __init__(self, def: Node):
# Here we read and parse all the settings from the elements
self.arches = def["supported_arches"]
self.kernel = def["kernel"]
self.extra_cmdline = def["append_cmdline"]
self.baseline_packages = def["baseline_packages"] # This gets set
project-wide
self.extra_packages = def["extra_packages"] # This gets set per-image
[...]
def generate(self) -> Dict[str, Node]:
arch_check = [f"arch == {supported}" for supported in
self.arches].join(" || ");
# Here we return all of the elements that I had the "codegen"
script generating before
return {
# The `stack` that pulls in all the dependencies
'pkgs.bst': Node({
'kind': 'stack',
'(?)': [ { arch_check: { '(!)': f"This device family only
supports {self.arches}" } } ]
'depends': self.baseline_packages + self.extra_packages,
}),
# The `compose` that generates /usr, using the stack defined above
'compose.bst': Node({
'kind': 'compose',
'dependencies': [ self._path_to_element('pkgs.bst') ],
[...]
}),
# Kernel, UKI, SBOM, final image build, etc etc etc
[...]
}
```
Here's another interesting example, that's not part of the motivating
use case but is still enabled by this architecture. Right now, it's
pretty painful to split subpackages out of a package (trying to keep
lists dependencies up-to-date, etc). We could use a generator to make
life easier:
```
# elements/sdk-deps/systemd.bstg
kind: subpackage
base:
kind: meson
sources: [...]
[...]
# Dependencies for all subpackages
depends:
- freedesktop-sdk.bst:bootstrap/glibc.bst
# Dependencies needed to build the package. Auto-extended by the
# generator with the runtime dependencies of all subpackages
build-depends:
- [...]
subpackages:
libs:
files:
- '%{usr}/lib/libsystemd.so'
- '%{usr}/lib/libsystemd-shared.so'
bootable:
files:
- '%{usr}/lib/systemd/systemd'
- [...]
depends:
- sdk-deps/systemd.bstg/libs.bst
- freedesktop-sdk.bst:components/kmod.bst
- [...]
initrd: [...]
```
```
# plugins/generators/subpackage.py
class Subpackage(Generator):
def __init__(self, def: Node):
self.base = def["base"]
self.subpackages = def["subpackages"]
def generate(self) -> Dict[str, Node]:
# Extend the split_rules of the base with a new rule for each
subpackage
rules = self.base["public"]["bst"]["split_rules"]
for name, opts in self.subpackages:
rules["__subpackage_" + name] = opts["files"]
self.base["public"]["bst"]["split_rules"] = rules
# Extend the build dependencies of the base w/ runtime deps of
subpackages
deps = self.base["build-depends"]
for name, opts in self.subpackages:
deps += opts["depends"]
# Filter out dependencies on our own subpackages
deps.filter(lambda dep: !dep.starts_with(self._path_to_generator()))
self.base["build-depends"] = deps
out = { 'base.bst': self.base }
for name, opts in self.subpackages:
out[subpackage + '.bst'] = Node({
'kind': 'filter',
'depends': opts["depends"] + self.base["depends"],
'build-depends': [ self._path_to_element('base.bst') ],
'config': {
'include': [ '__subpackage_' + name ],
}
})
return out
```
Anyway, I hope that paints a clear enough picture of how the feature
would work. Please let me know what you think, and what kind of things
I'd need to do before attempting an implementation of this.
Best,
Adrian
---
[1]: https://source.android.com/docs/setup/create/new-device#build-a-product
[2]: (An example Android definition for a real device)
https://android.googlesource.com/device/google/akita/+/refs/heads/main/device-akita.mk
[3]:
https://www.chromium.org/chromium-os/developer-library/guides/chromiumos-board-porting-guide/
[4]: (An example for a real ChromeOS device)
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/baseboard-geralt/
and
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/overlay-geralt/
[5]: https://wiki.postmarketos.org/wiki/Deviceinfo_reference
[6]: (An example postmarketOS device)
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/tree/master/device/community/device-oneplus-enchilada
[7]: https://gitlab.com/carbonOS/build-meta/-/blob/main/tools/codegen
[8]: https://gitlab.com/carbonOS/build-meta/-/blob/main/boards/README.md
P.S.: I'm sorry if this shows up multiple times on the mailing list.
I've been struggling to get this email posted onto there. I couldn't
find any documentation about the requirements for posting to this list;
this is my third attempt to send, after subscribing and also confirming
my subscription.
Re: New type of plugin that generates multiple elements
Posted by Adrian Vovk <ad...@gmail.com>.
Hi all,
After a little bit of thinking about a conversation off list, and also
Nathan's idea, I have a possible alternate approach for my use-case.
Each board could be its own distinct bst project (potentially in its own
repo), that pulls in upstream GNOME OS through a junction. We can then
use the junction's element replacement feature to inject whatever we
need into the build pipeline. For example, we'd define some integration
points:
```
# gnomeos/elements/board/kernel.bst
# This is a stub element
kind: stack
# We can't build GNOME OS without a kernel. So, if the board doesn't
provide a kernel to use, make the build fail!
(!): "The board must define a kernel!"
```
```
# gnomeos/elements/board/extra-packages.bst
kind: stack
# This is a stub element, but it's a complete no-op. Does nothing unless
overridden
```
Anyway, you'd have a ton of these stub elements. Then the rest of the
build pipeline can be defined in terms of these stubs. For example,
there might be an elements/board/_internal/uki.bst that depends on
`board/kernel.bst` and `board/extra-kernel-cmdline.bst` and feeds their
generated artifacts into ukify to make a bootable kernel UKI with a
kernel command-line baked in. Ultimately you end up with an
elements/image.bst, that depends on all the elements/board/_internal/
stuff, and spits out the final disk image. In the GNOME OS repo, if you
try to `bst build image.bst` you'll get failures because of all the stub
elements.
Then, each board gets a bst project. The microsoft-surface board I
describe above, for instance, could look like:
- project.conf
- elements/
- image.bst
- gnome.bst
- linux-surface.bst
- kernel-cmdline.bst
- extra-packages.bst
- webcam-firmware.bst
- extra-power-profiles.bst
- detatchable-keyboard-daemon.bst
The important element here is gnome.bst, which is our junction:
```
# elements/gnome.bst
kind: junction
source:
- ...
config:
options:
arch: "%{arch}" # Selectable arch, because Surface devices come in
x86_64 and aarch64
replace:
board/kernel.bst: linux-surface.bst
board/extra-packages.bst: extra-packages.bst
board/extra-kernel-cmdline.bst: kernel-cmdline.bst
```
Then, for CLI convenience, we could define `image.bst` as an alias:
```
kind: link
config:
target: gnome.bst:image.bst
```
And that's that! `bst build image.bst` in the microsoft-surface bst
project will give you a packaged up GNOME OS image for that device.
Pros of this approach:
- Works with existing buildstream
- We can pretty easily experiment with various board enablement
out-of-tree (potentially privately) before upstreaming it
- Distinct per-device CI pipelines and rules
- You can replace arbitrary packages if necessary. Maybe some board
needs a custom build of mesa, then we can package that in the board's
own project and simply add it to our list of replacements
Cons of this approach:
- Requires an external project, which makes it kinda awkward to put in a
monorepo
- Lots of bst boilerplate: project.conf, aliases, plugins, mirrors, etc.
Need to figure out a good way to share this. A lot of it can probably be
pulled through the junction, but you still need the boilerplate to do that
I think this is a not-bad approach overall. Still, I believe that
generator plugins would be pretty convenient and nice-to-have to address
other usecases I mentioned (subpackages, for instance). Would love to
hear your thoughts.
Best,
Adrian