You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Adrian Vovk <ad...@gmail.com> on 2025/02/17 00:08:38 UTC

New type of plugin that generates multiple elements

Hi everyone,

I've got a proposal for some new functionality in BuildStream. Forgive 
me for the long email... I'm told I need to email here before I try 
writing any code. So, I'd like to first motivate the new feature, and 
then explain what I think it should look like.

# Background

One of my strategic goals for carbonOS (and now GNOME OS) has been to 
support device-specific images wherever appropriate. Device-specific 
images are, by nature, all very similar to each other: just some extra 
packages here, some partition table tweaks there, and so on. Build tools 
that deal with this (for example: Android's[1][2], ChromeOS's[3][4], and 
postmarketOS's[5][6]) have all converged on similar solutions. They all 
have a unified and highly parameterized image build pipeline, and 
device-specific configuration files that control the pipeline's numerous 
settings.

Buildstream, as it currently is, isn't well suited to create this 
optimal image-build pipeline. But why not? Well, composing a collection 
of buildstream elements into a single OS image generally requires more 
than one element definition: you need at least one `stack` to group up 
the packages that'll go into your image, then you need a 
`collect-manifest` to generate your SBOM, maybe a couple of `compose` 
elements to build your initrd and root directory trees, and finally some 
`script`s to shove everything into the final disk images. This is 
perfectly fine if you're building one image, but if you want to build a 
second device-specific variant you need to copy and then edit these 
elements. This quickly becomes unmaintainable. There's no mechanism in 
Buildstream to parameterize across element boundaries like we'd need to 
here.

My solution for this in carbonOS was a workaround. I added a "code 
generation" script [7], that parses a device-specific configuration file 
[8] and generates the necessary per-device elements for me. The bash 
script was definitely crude, but it did give me the solution I wanted: 
For each device, I could write a config file along with some 
device-specific package definitions, while reusing the behavior of a 
unified image build pipeline.

# Proposed Solution

I would like to introduce a new type of plugin to BuildStream, which I'm 
tentatively calling a "generator". Much like an element, each instance 
of a generator plugin corresponds to a generator definition file 
(.bstg). Much like an element, we decide which plugin to instantiate 
from the `kind:` field found in a generator definition file. A 
generator's job would be to transform the contents of its definition 
file into a dict of `Node`s that can then be parsed into BuildStream 
elements as if they were read from disk.

Each entry in the dict maps a filename to the root `Node` of the 
corresponding generated element. This allows other elements, from 
outside of the generator, to refer to elements created by the generator. 
For example, let's say there's a generator defined at 
elements/something/demo.bstg, which generates two elements: foo.bst and 
bar.bst. Elements anywhere in the project can refer to these generated 
elements via something/demo.bstg/foo.bst and something/demo.bstg/bar.bst 
respectively. From the perspective of other elements, a generator 
appears like a directory that contains normal element definitions inside.

I think the easiest way to explain how this generator system would work 
in practice would be via examples. So let's do that. Here's what the 
simplest generator plugin might look like (pseudocode):

```
# plugins/generators/basic.py
class BasicGenerator(Generator):
    def __init__(self, def: Node):
       self.elements = def["elements"]

    def generate(self) -> Dict[str, Node]:
       return self.elements
```

This generator just lets you define multiple elements in one file. It 
does nothing special other than that. A corresponding definition file 
might look like this:

```
# elements/demo.bstg
kind: basic

elements:
   foo.bst:
     kind: meson

     [...]

   foo-filtered.bst:
     kind: filter

     build-depends:
     - demo.bstg/foo.bst

     [...]
```

Of course, the point of these generator plugins would be to do something 
more interesting. So, here's an example of what a generator implementing 
our device-specific image build pipeline would look like:

```
# elements/boards/microsoft-surface/board.bstg
kind: device
description: Board definition for the Microsoft Surface series of devices

supported_arches:
- x86_64 # Most devices before 2024
- aarch64 # Devices after 2024 switched to Arm

kernel: boards/microsoft-surface/linux-surface.bst

extra_packages:
- boards/microsoft-surface/webcam-firmware.bst
- boards/microsoft-surface/extra-power-profiles.bst
- boards/microsoft-surface/detatchable-keyboard-daemon.bst
- [...]

append_cmdline: >-
   intel.iommu=force
   qcom.iommu=enable

[...]
```

```
# plugins/generators/device.py
class Device(Generator):
    def __init__(self, def: Node):
       # Here we read and parse all the settings from the elements
       self.arches = def["supported_arches"]
       self.kernel = def["kernel"]
       self.extra_cmdline = def["append_cmdline"]
       self.baseline_packages = def["baseline_packages"] # This gets set 
project-wide
       self.extra_packages = def["extra_packages"] # This gets set per-image
       [...]

    def generate(self) -> Dict[str, Node]:
       arch_check = [f"arch == {supported}" for supported in 
self.arches].join(" || ");

       # Here we return all of the elements that I had the "codegen" 
script generating before
       return {
          # The `stack` that pulls in all the dependencies
          'pkgs.bst': Node({
             'kind': 'stack',
             '(?)': [ { arch_check: { '(!)': f"This device family only 
supports {self.arches}" } } ]
             'depends': self.baseline_packages + self.extra_packages,
          }),

          # The `compose` that generates /usr, using the stack defined above
          'compose.bst': Node({
             'kind': 'compose',
             'dependencies': [ self._path_to_element('pkgs.bst') ],
             [...]
          }),

          # Kernel, UKI, SBOM, final image build, etc etc etc
          [...]
       }
```

Here's another interesting example, that's not part of the motivating 
use case but is still enabled by this architecture. Right now, it's 
pretty painful to split subpackages out of a package (trying to keep 
lists dependencies up-to-date, etc). We could use a generator to make 
life easier:

```
# elements/sdk-deps/systemd.bstg
kind: subpackage

base:
   kind: meson
   sources: [...]
   [...]

   # Dependencies for all subpackages
   depends:
   - freedesktop-sdk.bst:bootstrap/glibc.bst

   # Dependencies needed to build the package. Auto-extended by the
   # generator with the runtime dependencies of all subpackages
   build-depends:
   - [...]

subpackages:
   libs:
     files:
     - '%{usr}/lib/libsystemd.so'
     - '%{usr}/lib/libsystemd-shared.so'

   bootable:
     files:
     - '%{usr}/lib/systemd/systemd'
     - [...]
     depends:
     - sdk-deps/systemd.bstg/libs.bst
     - freedesktop-sdk.bst:components/kmod.bst
     - [...]

   initrd: [...]
```

```
# plugins/generators/subpackage.py
class Subpackage(Generator):
    def __init__(self, def: Node):
       self.base = def["base"]
       self.subpackages = def["subpackages"]

    def generate(self) -> Dict[str, Node]:
       # Extend the split_rules of the base with a new rule for each 
subpackage
       rules = self.base["public"]["bst"]["split_rules"]
       for name, opts in self.subpackages:
          rules["__subpackage_" + name] = opts["files"]
       self.base["public"]["bst"]["split_rules"] = rules

       # Extend the build dependencies of the base w/ runtime deps of 
subpackages
       deps = self.base["build-depends"]
       for name, opts in self.subpackages:
          deps += opts["depends"]
       # Filter out dependencies on our own subpackages
       deps.filter(lambda dep: !dep.starts_with(self._path_to_generator()))
       self.base["build-depends"] = deps

       out = { 'base.bst': self.base }
       for name, opts in self.subpackages:
          out[subpackage + '.bst'] = Node({
              'kind': 'filter',
              'depends': opts["depends"] + self.base["depends"],
              'build-depends': [ self._path_to_element('base.bst') ],
              'config': {
                 'include': [ '__subpackage_' + name ],
              }
          })
       return out
```

Anyway, I hope that paints a clear enough picture of how the feature 
would work. Please let me know what you think, and what kind of things 
I'd need to do before attempting an implementation of this.

Best,
Adrian

---

[1]: https://source.android.com/docs/setup/create/new-device#build-a-product
[2]: (An example Android definition for a real device) 
https://android.googlesource.com/device/google/akita/+/refs/heads/main/device-akita.mk
[3]: 
https://www.chromium.org/chromium-os/developer-library/guides/chromiumos-board-porting-guide/
[4]: (An example for a real ChromeOS device) 
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/baseboard-geralt/ 
and 
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/refs/heads/main/overlay-geralt/
[5]: https://wiki.postmarketos.org/wiki/Deviceinfo_reference
[6]: (An example postmarketOS device) 
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/tree/master/device/community/device-oneplus-enchilada
[7]: https://gitlab.com/carbonOS/build-meta/-/blob/main/tools/codegen
[8]: https://gitlab.com/carbonOS/build-meta/-/blob/main/boards/README.md

P.S.: I'm sorry if this shows up multiple times on the mailing list. 
I've been struggling to get this email posted onto there. I couldn't 
find any documentation about the requirements for posting to this list; 
this is my third attempt to send, after subscribing and also confirming 
my subscription.

Re: New type of plugin that generates multiple elements

Posted by Adrian Vovk <ad...@gmail.com>.
Hi all,

After a little bit of thinking about a conversation off list, and also 
Nathan's idea, I have a possible alternate approach for my use-case.

Each board could be its own distinct bst project (potentially in its own 
repo), that pulls in upstream GNOME OS through a junction. We can then 
use the junction's element replacement feature to inject whatever we 
need into the build pipeline. For example, we'd define some integration 
points:

```
# gnomeos/elements/board/kernel.bst
# This is a stub element
kind: stack

# We can't build GNOME OS without a kernel. So, if the board doesn't 
provide a kernel to use, make the build fail!
(!): "The board must define a kernel!"
```

```
# gnomeos/elements/board/extra-packages.bst
kind: stack
# This is a stub element, but it's a complete no-op. Does nothing unless 
overridden
```

Anyway, you'd have a ton of these stub elements. Then the rest of the 
build pipeline can be defined in terms of these stubs. For example, 
there might be an elements/board/_internal/uki.bst that depends on 
`board/kernel.bst` and `board/extra-kernel-cmdline.bst` and feeds their 
generated artifacts into ukify to make a bootable kernel UKI with a 
kernel command-line baked in. Ultimately you end up with an 
elements/image.bst, that depends on all the elements/board/_internal/ 
stuff, and spits out the final disk image. In the GNOME OS repo, if you 
try to `bst build image.bst` you'll get failures because of all the stub 
elements.

Then, each board gets a bst project. The microsoft-surface board I 
describe above, for instance, could look like:

- project.conf
- elements/
   - image.bst
   - gnome.bst
   - linux-surface.bst
   - kernel-cmdline.bst
   - extra-packages.bst
   - webcam-firmware.bst
   - extra-power-profiles.bst
   - detatchable-keyboard-daemon.bst

The important element here is gnome.bst, which is our junction:

```
# elements/gnome.bst
kind: junction

source:
- ...

config:
   options:
     arch: "%{arch}" # Selectable arch, because Surface devices come in 
x86_64 and aarch64
   replace:
     board/kernel.bst: linux-surface.bst
     board/extra-packages.bst: extra-packages.bst
     board/extra-kernel-cmdline.bst: kernel-cmdline.bst
```

Then, for CLI convenience, we could define `image.bst` as an alias:

```
kind: link
config:
    target: gnome.bst:image.bst
```

And that's that! `bst build image.bst` in the microsoft-surface bst 
project will give you a packaged up GNOME OS image for that device.

Pros of this approach:
- Works with existing buildstream
- We can pretty easily experiment with various board enablement 
out-of-tree (potentially privately) before upstreaming it
- Distinct per-device CI pipelines and rules
- You can replace arbitrary packages if necessary. Maybe some board 
needs a custom build of mesa, then we can package that in the board's 
own project and simply add it to our list of replacements

Cons of this approach:
- Requires an external project, which makes it kinda awkward to put in a 
monorepo
- Lots of bst boilerplate: project.conf, aliases, plugins, mirrors, etc. 
Need to figure out a good way to share this. A lot of it can probably be 
pulled through the junction, but you still need the boilerplate to do that

I think this is a not-bad approach overall. Still, I believe that 
generator plugins would be pretty convenient and nice-to-have to address 
other usecases I mentioned (subpackages, for instance). Would love to 
hear your thoughts.

Best,
Adrian