You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Benjamin Schubert <co...@benschubert.me> on 2020/10/09 14:24:19 UTC
Conditionally open the network in the build sandbox
Hey everyone,
TLDR: I would like a way of allowing network access during builds on a
per-element basis, to ease transition into BuildStream.
I realize this has a potential to be highly controversial, but I think that
the benefits in terms of ease of adoption makes it worth it.
Problem
-----------
In some cases, it is very hard to completely sandbox a build, with
some build systems absolutely wanting to go to the network. There are
workaround to go over those, however, having to completely fix them before
moving sizeable projects to BuildStream makes it very hard to adopt.
In other cases, you might want access to network resources for testing elements
that would require some network access
I know and fully understand that having network access during a sandbox build
removes many of the benefits of BuildStream (source caching, repeatable builds),
but those can be addressed at an infrastructure level before being moved to
work as it should in BuildStream.
How would that look like?
--------------------------
I would envision a new variable option, similar to 'max-jobs', that would be
transparent to the plugin, but used by BuildStream to open network access.
How much work would we need to do for this?
------------------------------------------
We already have mechanisms to open the sandbox when running a shell.
We would need to make this possible for regular builds.
This would also mean we need to be able to be able to set it on platform
properties for remote execution.
Mitigations
-----------
Some potential mitigations around this being 'bad' could be:
- Optionally disable by default the pushing of such artifacts to remote caches,
like we currently handle workspaces, with the ability to enable it either at
the project level or the element level.
Any thoughts?
Thanks!
Ben
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi,
On Wed, 2020-10-28 at 18:19 +0000, Benjamin Schubert wrote:
> Hey,
>
> Sorry for the late response. Responses inline
>
> The TLDR would be that we cannot have user interaction and, in some cases it
> is not possible to know the dependencies before executing the code.
> We need to have this intermediate state of the system to prove it could work
> before moving to it and actually being able to restrict access to the sandbox.
>
Alright, you've made fair points in this thread, I also think we had
discussed a long time ago that we should allow the user to shoot
themselves in the foot if they so desire (Sander phrases this as
"giving them enough rope to hang themselves with").
Let's make the warning signs around this as loud as possible.
It would be nice for instance to have a summary (unconditional warning)
at load time of all elements which bought into this backdoor.
There are some other parts of this I still think are pertinent to
discuss, regardless of the bottom line of the proposal.
Consider the other replies inline to not have baring over the outcome
of the proposal...
[...]
> > This is after all why we developed plugins like the cargo plugin for
> > rust apps (some weird languages want to download dependencies from the
> > internet) - if we can write source plugins for these problematic
> > languages, this is a better avenue to pursue than to just give up and
> > open up the network at build time.
>
> This is correct, however, it is not always possible to make it entirely
> automatic without user interactions.
> For large organizations, it is much better to be able to demonstrate that a
> system could work without user interactions first, and then moving the system
> up from a set point, therefore stopping regressions and progressively making
> the whole system better.
With the 'pip' plugin we have, we it should essentially only require an
initial run of `bst track` after appending the `pip` source to your
python build, so as to pick up the python dependencies and encode the
specific versions into your project.
I don't think this really counts as human actions any more than
migrating a build to use BuildStream otherwise does.
Maybe you are referring to other human actions I'm overlooking ?
>
> > Do we have concrete examples of workflows which are difficult to port
> > which cannot be solved with source plugins we can write ?
>
> The current `pip` source plugin (and friends) assume that the set of
> dependencies to install will be the same same regardless of which platform pip
> is running on and the python version that is used. This is not true as there
> are platform and version specific dependencies, etc.
This statement raises concern (and is the main reason for my elaborate
reply to this email).
If the pip plugin relies too heavily on host tooling to make invalid
inferences about what should be downloaded, this is a serious problem.
The pip plugin is supposed to simply encode up to date versions of
python dependencies which match up with the project's requirements at
track time, and then download those at build time.
If this behavior can vary depending on host pip, then that is a serious
bug in the `pip` plugin implementation - and probably needs to be
implemented differently (possibly without leveraging the `pip` code
itself at all).
> In more general terms, language-specific building tools that act as dependency
> managers and download the dependency based on the host platform they are
> running from. Again, this would not be a problem with user interaction and
> forcing one way of doing. However that is not an option for us.
This is certainly not how source transforms developed for such
languages are intended to work, whether we leverage that host
language's package manager code or not to implement these plugins; it
is our responsibility to ensure that our plugins which do the "source
transform" thing do so reliably consistently regardless of the host
system.
If target system comes into play as a matter of necessity (i.e. the
correct code cannot be inferred without knowing what kind of system the
code should be built/run on) - then this information needs to be
encoded into the Source configuration.
E.g. the `pip` source needs to have configurations stating things like
the python version expected to be in use in the sandbox, if the python
version in use makes a difference, and then the plugin has to "just do
the right thing".
The idea that it is acceptable for the pip plugin to behave in a host
dependent way is rather frightening, and we certainly should not bless
the plugin upstream without it passing these basic requirements first.
> > [...]
> > > > If network access were "supported", this could lead us down a road
> > > > where we are requested to rebuild things when artifacts are too old
> > > > (for instance, I cannot build against this dependency, because the
> > > > random source code it downloaded from the internet only 3 months ago is
> > > > no longer valid for me to build against).
> > >
> > > Again, if you download from the internet that is a problem. If you have already
> > > a cache for such dependencies in your organization, that is not a concern.
> >
> > I think the above is dangerous thinking, and there are certainly still
> > concerns.
> >
> > Just because you are maintaining a store of sources in your own
> > organization does not really mean that those sources are not random,
> > and while it might work for a time (or even forever, if you are very
> > careful to always use it correctly), it will be difficult to maintain
> > and guarantee correctness of your builds over longer periods of time.
> >
> > Essentially, this externalizes the problem of matching up a specific
> > set of inputs with a given build, making it the organizations problem
> > instead of using BuildStream to specifically address all inputs (with
> > checksums or git shas or the like)... I can see this being a cause of
> > frustration years down the line.
>
> [snip]
>
> > This margin of error can be eliminated by ensuring that your
> > BuildStream project data addresses sources specifically and does not
> > tolerate variants (like tarball address without checksum for instance).
>
> I am not saying this is perfect.
To clarify, I was mostly speaking here to the "that is not a concern"
portion of your previous statement about having sources cached within
the organization.
Such a statement can be dangerous inasmuch as one can interpret that
and then disregard the finer implications that one needs to maintain
the association between the versions of your project which must match
the versions of your cached sources.
Cheers,
-Tristan
Re: Conditionally open the network in the build sandbox
Posted by Benjamin Schubert <co...@benschubert.me>.
Hey,
Sorry for the late response. Responses inline
The TLDR would be that we cannot have user interaction and, in some cases it
is not possible to know the dependencies before executing the code.
We need to have this intermediate state of the system to prove it could work
before moving to it and actually being able to restrict access to the sandbox.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, October 17th, 2020 at 9:14 AM, Tristan Van Berkom <tr...@codethink.co.uk> wrote:
[snip]
> >
> > The point is about being able to incrementally improve the way your builds are
> > made. It can be extremely hard, or too costly for a company to move everything
> > to a 'correct' point before being able to move to BuildStream.
> >
> > This opens more an easy adoption of BuildStream, by allowing most of the
> > benefits of it, and allowing to improve the 'bad' elements later, rather than
> > having a way of solving every problem before moving to it.
> >
> > For me, this is a key point that BuildStream offered, compared to, for example,
> > Bazel, which is more or less 'all or nothing'.
> >
> > Yes, having network access during your builds can be bad, and we should
> > absolutely discourage that. However, I strongly believe that in order to help
> > with adoption, users should be allowed to explicitely ask the permission to
> > shoot themselves in the foot.
>
> I'm not trying to block this outright, but still want to caution that
> this really might do more harm than it does good.
>
> Also, I would be more interested to see examples which are specifically
> difficult to deal with, and work on making those use cases easier to
> deal with.
The main problem I see is around languages' build systems that want to act as
package managers very strongly. There is not always a good and easy way of
downloading the correct dependencies (npm/python are two examples there),
which would usually work if we got human actions converting the projects.
When wanting to move a lot of projects in one go, it would be far easier for us
to be able to move to BuildStream first, and then handle the hermeticity as a
separate work stream. Essentially detangling the move to the new 'system'
from some of the underlying infrastructure work that is required.
> This is after all why we developed plugins like the cargo plugin for
> rust apps (some weird languages want to download dependencies from the
> internet) - if we can write source plugins for these problematic
> languages, this is a better avenue to pursue than to just give up and
> open up the network at build time.
This is correct, however, it is not always possible to make it entirely
automatic without user interactions.
For large organizations, it is much better to be able to demonstrate that a
system could work without user interactions first, and then moving the system
up from a set point, therefore stopping regressions and progressively making
the whole system better.
>
> Do we have concrete examples of workflows which are difficult to port
> which cannot be solved with source plugins we can write ?
The current `pip` source plugin (and friends) assume that the set of
dependencies to install will be the same same regardless of which platform pip
is running on and the python version that is used. This is not true as there
are platform and version specific dependencies, etc.
In more general terms, language-specific building tools that act as dependency
managers and download the dependency based on the host platform they are
running from. Again, this would not be a problem with user interaction and
forcing one way of doing. However that is not an option for us.
>
> [...]
> > > If network access were "supported", this could lead us down a road
> > > where we are requested to rebuild things when artifacts are too old
> > > (for instance, I cannot build against this dependency, because the
> > > random source code it downloaded from the internet only 3 months ago is
> > > no longer valid for me to build against).
> >
> > Again, if you download from the internet that is a problem. If you have already
> > a cache for such dependencies in your organization, that is not a concern.
>
> I think the above is dangerous thinking, and there are certainly still
> concerns.
>
> Just because you are maintaining a store of sources in your own
> organization does not really mean that those sources are not random,
> and while it might work for a time (or even forever, if you are very
> careful to always use it correctly), it will be difficult to maintain
> and guarantee correctness of your builds over longer periods of time.
>
> Essentially, this externalizes the problem of matching up a specific
> set of inputs with a given build, making it the organizations problem
> instead of using BuildStream to specifically address all inputs (with
> checksums or git shas or the like)... I can see this being a cause of
> frustration years down the line.
[snip]
>
> This margin of error can be eliminated by ensuring that your
> BuildStream project data addresses sources specifically and does not
> tolerate variants (like tarball address without checksum for instance).
I am not saying this is perfect. Technically I would prefer having everything
handled by BuildStream, but big systems need a way forward that does not
require:
- A flag day change
- A lot of pre-work before showing it can work
>
> > > Mostly I worry that if we give users any opportunity to allow internet
> > > access in builds, we almost guarantee that they will not do the minimal
> > > legwork required to ensure their build works without network access.
> >
> > I believe that this is to the users to decide. We can make it very clear
> > in the docs that this is a bad idea, and users can decide to shoot themselves
> > in the foot.
> >
> [...]
> > No, again, this is a documentation concern. We can document that by using this
> > you understand your element is not repeatable etc.
> >
> > Fixing 'one element' is something that is doable indeed. Doing that at a
> > company scale, on large projects become extremely hard. Being able to move to
> > BuildStream before, allows to stop the bleeding for new elements and cleanup
> > the rest on a longer term.
>
> I am worried about underestimating the willingness of users to not fix
> a problem when given any opportunity, I expect that given a simple
> switch, people will always turn it on at any time any element gives
> them trouble even if it was relatively easy to fix, and they will not
> circle back to fix this problem element until it has hurt them, twice.
>
> Regardless of whether we warned users in advance, they would be right
> to blame us for not forcing them to do a small amount of work in
> advance and save them from possible problems down the line.
I understand your concerns, and I would much prefer being able to move forward
without requiring such options. However, I still think that users are responsible
in the end on how to setup their system. In the end, they could also simply
use 'buildbox-run-hostools' to use BuildStream and not gain 90% of the safety
that BuildStream provides (which I am not advocating for).
>
> In conclusion, these are my general feelings on the matter:
>
> * If there are cases I've overlooked where a "source transform"
> would not remedy the situation, then maybe we need a switch
> like you suggest, but I'd really like to be enlightened as to
> what those cases are.
Those systems usually require a user interaction to enable the source transform
to work correctly. That's sadly not always possible.
>
> * Right now we only have "source transforms" for python and rust
> (the pip and the cargo sources).
>
> This is a sad state of affairs, there are probably some other
> weird languages doing similar things, node.js comes to mind.
>
> Is it realistic for us to be able to cover every language which
> downloads external dependencies from the internet at build time
> with a source plugin (like pip/cargo) that we maintain in our
> trusted plugin collection ?
>
> I feel like this is realistic, but if not, then maybe it also
> calls for such a switch.
I agree this is realistic and would like to work toward that. I however don't
think this removes the need for the switch in some edge cases.
>
> * There are some cases which are ridiculously easy to fix, like
> scripts which want to download config.sub and config.guess from
> git.savannah.gnu.org (which sometimes need to be fixed to copy them
> in from /usr/share/automake/ instead), or builds like cracklib
> which might want to download it's word dictionary if the user
> forgot to separately put it in place.
>
> In these cases, I worry that providing a switch is going to
> result in most of these easy to fix issues being completely
> ignored and worked around, leading to a generally bad user
> experience down the road.
>
>
> Cheers,
> -Tristan
Hope that helps,
Cheers,
Ben
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi,
On Tue, 2020-10-27 at 19:28 +0000, Chandan Singh wrote:
[...]
> The shell functionality already allows network access for purposes
> like this, and comes in two variants - build shell and runtime shell.
>
> The build shell will have all the build dependencies and sources for
> the element in exactly the same way that they would be staged during
> a
> `bst build`. It can be used to debug builds of the project, for
> example, to iterate on various configuration options.
>
> The runtime shell will provide access to the built artifacts of the
> element and all its runtime dependencies. As such, it is more useful
> for testing the resulting binary.
>
> Both the shells allow network access. As such, they are not meant to
> produce deployable artifacts, but can be very useful for debugging
> purposes. There are probably still many things that could be improved
> in the shell experience. So, I'd be very interested for you to try
> the
> shell functionality and let us know if that works for your use case.
>
> One issue that I can already think of is that if we go with this
> approach, you will also need to have a debugger available in the
> BuildStream project and somehow add a dependency on it for invoking
> the build shell. This topic has been a recurring discussion item, but
> we haven't solved it yet.
> https://gitlab.com/BuildStream/buildstream/-/issues/422 has some
> related background on this.
>
> A couple of references below.
>
> CLI docs:
> https://docs.buildstream.build/master/using_commands.html#invoking-shell
> Shell in action:
> https://docs.buildstream.build/master/tutorial/running-commands.html#run-the-hello-world-program
>
To really make good use of the shell, I should point out that there are
some additional holes you might want to poke in the sandbox, which
BuildStream explicitly supports doing in order to test built artifacts
in your host environment, with the assumption that your host machine
somewhat resembles the target environment where the artifact is
expected to actually run, this is done through the `shell` project
configuration:
https://docs.buildstream.build/master/format_project.html#customizing-the-shell
Currently it is possible to run your apps using the host X server and
the host pulseaudio, and even leverage graphics acceleration from the
host:
https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/project.conf#L161
It is of course important to test the deterministically built artifacts
BuildStream produces on real running systems, this is especially true
for the edit/compile/test cycle scenario.
This control over poking explicit holes in the sandbox has so far been
able to handle everything one could need on Linux (we could extend that
this turns out to be untrue, and will probably need to extend this
configuration when adding support for exotic platforms like windows).
Cheers,
-Tristan
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Justin,
On Thu, 2020-12-31 at 12:19 -0500, Justin Erenkrantz wrote:
> On Tue, Dec 29, 2020 at 5:00 AM Tristan Van Berkom <
> tristan.vanberkom@codethink.co.uk> wrote:
>
[...]
> Yup, it was the lack of resolv.conf - thanks! I'll incorporate some of
> these tricks into serf-bst's project.conf. Does it make sense to have
> slightly saner defaults in buildstream?
Not really.
BuildStream itself is only a build tool and is as decoupled as possible
from the product being built with the tool.
As such, it is a goal for us to remove as much as we can from the
default project.conf settings before 2.0:
https://github.com/apache/buildstream/blob/master/src/buildstream/data/projectconfig.yaml
As you can see, most of the settings already present you are building a
GNU/Linux system, or something that will run on such a system.
We are hoping to remove anything that is payload specific from such
settings, so that these can be built from the ground up for any given
project and any OS we might support.
That said, this does not preclude sharing of default settings across
projects, for instance one could share defaults using cross junction
includes:
https://docs.buildstream.build/1.93.5/junctions/junction-includes.html
[...]
>
> Well, the issue is that there is a vicious cycle between gdb's source
> dependencies.
>
Cyclic dependencies are an unfortunate fact of life indeed :)
> I tried creating elements for gdb - but, it requires makeinfo to build due
> to this issue:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=14678
>
> Texinfo requires gettext which requires wget...which requires makeinfo to
> build.
>
> So, I can't technically define an upstream element for gdb from source.
> So, it must be injected outside Buildstream (which we have via our platform
> Docker image)...unless there's another trick to break these dependencies
> somehow?
With your current approach, you would want a gdb that is built for the
runtime which you are using, in which case indeed it would be best to
take the gdb which corresponds to the docker image you are based on.
This could be done with an additional import/filter element to select
only the gdb bits from your "platform Docker image".
Alternatively, you might consider basing your project on
freedesktop-sdk instead of an opaque docker image, or try building it
in a similar way as freedesktop-sdk is doing it:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/elements/components/gdb.bst
Interestingly, we do not build makeinfo with freedesktop-sdk, but we
also build it directly from git rather than a dist tarball.
That said, no matter how you fry the cat you still run into the issue
#422 I mentioned earlier.
I.e. you will have to either have the end result actually depend on gdb
or not, until we add a way to stage multiple elements/targets into the
same sandbox at `bst shell` time.
> Related to the above, is the still unresolved issue #413:
> > https://gitlab.com/BuildStream/buildstream/-/issues/413
> >
> > This pertains to adding build directories to sandbox for `bst shell`,
> > at the same locations where they were originally built, so that
> > debuggers can easily find the sources for the binaries they were built
> > for.
> >
>
> Also, it doesn't look like we're keeping the debug symbols around - so,
> even if I inject gdb into the platform, I don't have the debug symbols.
> I'll hunt around to figure out if there's an obvious approach. So, I'm not
> even getting to the point of having the symbols linking to the
> (non-existent) source files...
>
In your case, it may just be how the toolchain is configured in the
docker image you are extracting your runtime from. Given that you are
using a Docker base, I would not expect that base to include debug
symbols for the dependencies, but that does not necessarily mean you
would not have them for the libraries/apps which you are building.
More details
~~~~~~~~~~~~
If "strip-binaries" is not defined at all (which is default), then
compiled binaries do not have their debug info forcefully stripped from
them.
With fdsdk, it seems that they have baked something pretty custom for
some reason or another, but essentially the base toolchain is
generating output with debug information included, and this debug
information gets stripped out of the binaries and installed separately,
via the "%{strip-binaries}" variable declaration:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/include/strip.yml
The "freedesktop-sdk-stripper" program is built manually from the files
at:
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/tree/master/files/stripper
Normally this is done with a `find "%{install-root}" -exec ..` loop
which invokes `objdump` and puts the debug info into corresponding
files in "%{libdir}/debug".
> > Of course, when we start talking about graphical debuggers and IDEs
> > then this becomes a much more complex discussion.
> >
>
> One point Sander mentioned is that some of the IDEs support remote symbol
> serving - so, perhaps something we can explore that might ease the barrier
> to entry.
>
Exactly, this one line of progress we discussed, which is still quite
complex, as the symbols are inside the sandbox and need to be served
from the sandbox in such a way that the IDE can discover it.
It is a lot of legwork to come up with a design which makes it (fairly)
easy enough for projects to setup and work with, I suppose the first
step would be to create a hacky prototype for discovery purposes and
then design around that.
Cheers,
-Tristan
Re: Conditionally open the network in the build sandbox
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Tue, Dec 29, 2020 at 5:00 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Hi Justin,
>
> On Mon, 2020-12-28 at 14:29 -0500, Justin Erenkrantz wrote:
> > On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
> > wrote:
> [...]
> > Thanks. I'm picking this up again as there's an upstream change in
> OpenSSL
> > which is apparently breaking serf. For reference, my serf-bst 2.0
> project
> > is at [1].
> >
> > Buildstream is *so* close to doing what I want here: control my version
> of
> > OpenSSL and Serf in a clean manner and do so in an isolated manner from
> my
> > host environment.
>
> I took a look at your linked serf-bst project, and I feel like I am
> missing some context to be properly equipped to help more.
>
> What I am seeing is a docker used to create an input runtime upon which
> you are building some software, but I don't understand the context
> under which this is intended to be deployed or run.
>
The intention is to create the binary artifacts associated with the
dependency tree in a controlled manner.
Serf itself is just an HTTP library intended to be consumed by other
projects (such as Subversion, OpenOffice, etc, etc).
While there isn't a lot of activity in the project itself, there has been
recent churn in OpenSSL which silently breaks their API contracts. So,
having an integration build be able to easily swap out versions of OpenSSL
is very very useful.
> However, I need the debugger available to step through the test binaries
> to
> > figure out what's going on. It also seems like the bst shell doesn't
> have
> > network access - I can't run any of the produced binaries to remote
> hosts.
> > Is there a step I'm missing to allow network access? And, is the only
> way
> > to bring in the debugger is to build it into the base platform (or have a
> > special layer with gdb and such)?
>
> Regarding network access in `bst shell`
> ---------------------------------------
> It appears that network access should be enabled by default, and
> specifying the `--isolate` flag will ensure that there is no network
> access.
>
> That said, it is usually not enough to merely allow networking in order
> to have a networking enabled shell, to get networking to actually work,
> you will probably at least need to mount your host's /etc/resolv.conf
> into the shell environment.
>
> You can try this with:
>
> bst shell --mount /etc/resolv.conf /etc/resolv.conf serf.bst
>
> It is recommended to setup your preferred shell configuration in your
> project.conf to make things easier to run the sorts of software you
> build in a shell for testing/debugging:
>
> See the examples below for the `host-files` configuration:
>
> https://docs.buildstream.build/master/format_project.html#customizing-the-shell
>
> Here is a more elaborate example of a shell configuration which allows
> the user to launch graphics applications on (most) linux variants which
> use X (or XWayland) and pulseaudio:
>
>
> https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/project.conf#L165
Yup, it was the lack of resolv.conf - thanks! I'll incorporate some of
these tricks into serf-bst's project.conf. Does it make sense to have
slightly saner defaults in buildstream?
Regarding debugging tool presence in `bst shell`
> ------------------------------------------------
> This has been the subject of much debate a couple of years ago.
>
> In short: Yes, the only reliable way is to build your tools or wrap up
> some binaries known to work in that project's specific runtime
> environment and put them there, that means using elements.
>
> That however only answers half of the question when it comes to
> practical use of debuggers in `bst shell`, the other half is that a
> project does not truly depend on debuggers and profilers and such
> tooling.
>
> As far as I can see the conclusion of these debates is issue #422:
>
> https://gitlab.com/BuildStream/buildstream/-/issues/422
>
> This states that basically one should be allowed to stage multiple
> elements into the same `bst shell` environment, possibly resulting in
> issuing some file overlap warnings.
>
> This should not be all that difficult to implement, and implementing
> this would mean that one could have a defined set of debugging tools
> which one could optionally stage in `bst shell`.
>
Well, the issue is that there is a vicious cycle between gdb's source
dependencies.
I tried creating elements for gdb - but, it requires makeinfo to build due
to this issue:
https://sourceware.org/bugzilla/show_bug.cgi?id=14678
Texinfo requires gettext which requires wget...which requires makeinfo to
build.
So, I can't technically define an upstream element for gdb from source.
So, it must be injected outside Buildstream (which we have via our platform
Docker image)...unless there's another trick to break these dependencies
somehow?
Related to the above, is the still unresolved issue #413:
>
> https://gitlab.com/BuildStream/buildstream/-/issues/413
>
> This pertains to adding build directories to sandbox for `bst shell`,
> at the same locations where they were originally built, so that
> debuggers can easily find the sources for the binaries they were built
> for.
>
Also, it doesn't look like we're keeping the debug symbols around - so,
even if I inject gdb into the platform, I don't have the debug symbols.
I'll hunt around to figure out if there's an obvious approach. So, I'm not
even getting to the point of having the symbols linking to the
(non-existent) source files...
> Of course, when we start talking about graphical debuggers and IDEs
> then this becomes a much more complex discussion.
>
One point Sander mentioned is that some of the IDEs support remote symbol
serving - so, perhaps something we can explore that might ease the barrier
to entry.
Cheers. -- justin
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Justin,
On Mon, 2020-12-28 at 14:29 -0500, Justin Erenkrantz wrote:
> On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
> wrote:
[...]
> Thanks. I'm picking this up again as there's an upstream change in OpenSSL
> which is apparently breaking serf. For reference, my serf-bst 2.0 project
> is at [1].
>
> Buildstream is *so* close to doing what I want here: control my version of
> OpenSSL and Serf in a clean manner and do so in an isolated manner from my
> host environment.
I took a look at your linked serf-bst project, and I feel like I am
missing some context to be properly equipped to help more.
What I am seeing is a docker used to create an input runtime upon which
you are building some software, but I don't understand the context
under which this is intended to be deployed or run.
> However, I need the debugger available to step through the test binaries to
> figure out what's going on. It also seems like the bst shell doesn't have
> network access - I can't run any of the produced binaries to remote hosts.
> Is there a step I'm missing to allow network access? And, is the only way
> to bring in the debugger is to build it into the base platform (or have a
> special layer with gdb and such)?
Regarding network access in `bst shell`
---------------------------------------
It appears that network access should be enabled by default, and
specifying the `--isolate` flag will ensure that there is no network
access.
That said, it is usually not enough to merely allow networking in order
to have a networking enabled shell, to get networking to actually work,
you will probably at least need to mount your host's /etc/resolv.conf
into the shell environment.
You can try this with:
bst shell --mount /etc/resolv.conf /etc/resolv.conf serf.bst
It is recommended to setup your preferred shell configuration in your
project.conf to make things easier to run the sorts of software you
build in a shell for testing/debugging:
See the examples below for the `host-files` configuration:
https://docs.buildstream.build/master/format_project.html#customizing-the-shell
Here is a more elaborate example of a shell configuration which allows
the user to launch graphics applications on (most) linux variants which
use X (or XWayland) and pulseaudio:
https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/project.conf#L165
Regarding debugging tool presence in `bst shell`
------------------------------------------------
This has been the subject of much debate a couple of years ago.
In short: Yes, the only reliable way is to build your tools or wrap up
some binaries known to work in that project's specific runtime
environment and put them there, that means using elements.
That however only answers half of the question when it comes to
practical use of debuggers in `bst shell`, the other half is that a
project does not truly depend on debuggers and profilers and such
tooling.
As far as I can see the conclusion of these debates is issue #422:
https://gitlab.com/BuildStream/buildstream/-/issues/422
This states that basically one should be allowed to stage multiple
elements into the same `bst shell` environment, possibly resulting in
issuing some file overlap warnings.
This should not be all that difficult to implement, and implementing
this would mean that one could have a defined set of debugging tools
which one could optionally stage in `bst shell`.
Related to the above, is the still unresolved issue #413:
https://gitlab.com/BuildStream/buildstream/-/issues/413
This pertains to adding build directories to sandbox for `bst shell`,
at the same locations where they were originally built, so that
debuggers can easily find the sources for the binaries they were built
for.
Of course, when we start talking about graphical debuggers and IDEs
then this becomes a much more complex discussion.
Cheers,
-Tristan
Re: Conditionally open the network in the build sandbox
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
wrote:
> Exporting the build artifacts would certainly be one option. But
> depending on what you're trying to achieve, there may be other
> alternatives too. `bst shell` in particular comes to mind.
>
> The shell functionality already allows network access for purposes
> like this, and comes in two variants - build shell and runtime shell.
>
> The build shell will have all the build dependencies and sources for
> the element in exactly the same way that they would be staged during a
> `bst build`. It can be used to debug builds of the project, for
> example, to iterate on various configuration options.
>
> The runtime shell will provide access to the built artifacts of the
> element and all its runtime dependencies. As such, it is more useful
> for testing the resulting binary.
>
> Both the shells allow network access. As such, they are not meant to
> produce deployable artifacts, but can be very useful for debugging
> purposes. There are probably still many things that could be improved
> in the shell experience. So, I'd be very interested for you to try the
> shell functionality and let us know if that works for your use case.
>
> One issue that I can already think of is that if we go with this
> approach, you will also need to have a debugger available in the
> BuildStream project and somehow add a dependency on it for invoking
> the build shell. This topic has been a recurring discussion item, but
> we haven't solved it yet.
> https://gitlab.com/BuildStream/buildstream/-/issues/422 has some
> related background on this.
>
> A couple of references below.
>
> CLI docs:
> https://docs.buildstream.build/master/using_commands.html#invoking-shell
> Shell in action:
>
> https://docs.buildstream.build/master/tutorial/running-commands.html#run-the-hello-world-program
Thanks. I'm picking this up again as there's an upstream change in OpenSSL
which is apparently breaking serf. For reference, my serf-bst 2.0 project
is at [1].
Buildstream is *so* close to doing what I want here: control my version of
OpenSSL and Serf in a clean manner and do so in an isolated manner from my
host environment.
However, I need the debugger available to step through the test binaries to
figure out what's going on. It also seems like the bst shell doesn't have
network access - I can't run any of the produced binaries to remote hosts.
Is there a step I'm missing to allow network access? And, is the only way
to bring in the debugger is to build it into the base platform (or have a
special layer with gdb and such)?
I'm going to put this on pause for now and go back to fighting Debian sid
to see if I can get a reproducer...
Cheers. -- justin
1. https://github.com/jerenkrantz/serf-bst
Re: Conditionally open the network in the build sandbox
Posted by Chandan Singh <ch...@chandansingh.net>.
Hi Justin,
> From a development standpoint, we often need to be able to test against
> actual servers in case we can't reproduce over localhost. I guess that we
> could export the build artifacts from the sandbox and run them
> locally...I'd have to think about what the workflow would look like...we'd
> often want to jump into a debugger which needs the objects and source paths
> - so, I don't think it's necessarily straightforward. Or, am I missing a
> trick here?
Exporting the build artifacts would certainly be one option. But
depending on what you're trying to achieve, there may be other
alternatives too. `bst shell` in particular comes to mind.
The shell functionality already allows network access for purposes
like this, and comes in two variants - build shell and runtime shell.
The build shell will have all the build dependencies and sources for
the element in exactly the same way that they would be staged during a
`bst build`. It can be used to debug builds of the project, for
example, to iterate on various configuration options.
The runtime shell will provide access to the built artifacts of the
element and all its runtime dependencies. As such, it is more useful
for testing the resulting binary.
Both the shells allow network access. As such, they are not meant to
produce deployable artifacts, but can be very useful for debugging
purposes. There are probably still many things that could be improved
in the shell experience. So, I'd be very interested for you to try the
shell functionality and let us know if that works for your use case.
One issue that I can already think of is that if we go with this
approach, you will also need to have a debugger available in the
BuildStream project and somehow add a dependency on it for invoking
the build shell. This topic has been a recurring discussion item, but
we haven't solved it yet.
https://gitlab.com/BuildStream/buildstream/-/issues/422 has some
related background on this.
A couple of references below.
CLI docs: https://docs.buildstream.build/master/using_commands.html#invoking-shell
Shell in action:
https://docs.buildstream.build/master/tutorial/running-commands.html#run-the-hello-world-program
Cheers,
Chandan
Re: Conditionally open the network in the build sandbox
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Fri, Oct 23, 2020 at 1:46 PM Jürg Billeter <j...@bitron.ch> wrote:
> Hi Justin,
>
> On Fri, 2020-10-23 at 12:52 -0400, Justin Erenkrantz wrote:
> > How should downstream projects think about conducting build tests
> > (eg, 'make check', etc, etc)?
> >
> > In particular, we've been working on bringing Buildstream to Apache Serf
> > (which is an HTTP client library)[1]. Our current test suite requires
> > network access - it brings up a mock HTTP server and then runs test cases
> > with the local builds.
>
> A private loopback network is available in Bubblewrap sandboxes. I.e.
> as long as the mock HTTP server is brought up on localhost in the same
> BuildStream sandbox as the client tests, I expect this to work. Or have
> you encountered any issues with this?
>
> Issues related to running tests in BuildStream:
> * https://gitlab.com/BuildStream/buildstream/-/issues/776
> * https://gitlab.com/BuildStream/buildstream/-/issues/1327
Thanks for the pointers. When we have cycles to pick up the Buildstream
integration for serf, we'll keep these issues in mind.
From a development standpoint, we often need to be able to test against
actual servers in case we can't reproduce over localhost. I guess that we
could export the build artifacts from the sandbox and run them
locally...I'd have to think about what the workflow would look like...we'd
often want to jump into a debugger which needs the objects and source paths
- so, I don't think it's necessarily straightforward. Or, am I missing a
trick here?
Cheers. -- justin
Re: Conditionally open the network in the build sandbox
Posted by Jürg Billeter <j...@bitron.ch>.
Hi Justin,
On Fri, 2020-10-23 at 12:52 -0400, Justin Erenkrantz wrote:
> How should downstream projects think about conducting build tests
> (eg, 'make check', etc, etc)?
>
> In particular, we've been working on bringing Buildstream to Apache Serf
> (which is an HTTP client library)[1]. Our current test suite requires
> network access - it brings up a mock HTTP server and then runs test cases
> with the local builds.
A private loopback network is available in Bubblewrap sandboxes. I.e.
as long as the mock HTTP server is brought up on localhost in the same
BuildStream sandbox as the client tests, I expect this to work. Or have
you encountered any issues with this?
Issues related to running tests in BuildStream:
* https://gitlab.com/BuildStream/buildstream/-/issues/776
* https://gitlab.com/BuildStream/buildstream/-/issues/1327
Cheers,
Jürg
Re: Conditionally open the network in the build sandbox
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Sat, Oct 17, 2020 at 4:14 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Also, I would be more interested to see examples which are specifically
> difficult to deal with, and work on making those use cases easier to
> deal with.
>
> This is after all why we developed plugins like the cargo plugin for
> rust apps (some weird languages want to download dependencies from the
> internet) - if we can write source plugins for these problematic
> languages, this is a better avenue to pursue than to just give up and
> open up the network at build time.
>
> Do we have concrete examples of workflows which are difficult to port
> which cannot be solved with source plugins we can write ?
>
How should downstream projects think about conducting build tests (eg,
'make check', etc, etc)?
In particular, we've been working on bringing Buildstream to Apache Serf
(which is an HTTP client library)[1]. Our current test suite requires
network access - it brings up a mock HTTP server and then runs test cases
with the local builds.
Cheers. -- justin
1. See https://github.com/jerenkrantz/serf-bst for our current
proof-of-concept
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi,
On Fri, 2020-10-16 at 17:00 +0000, Benjamin Schubert wrote:
> Hey,
>
> Responses inline
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Wednesday, October 14th, 2020 at 11:45 AM, Tristan Van Berkom <tr...@codethink.co.uk> wrote:
>
> > Hi,
> >
> > On Fri, 2020-10-09 at 14:24 +0000,
> >
> > Benjamin Schubert wrote:
> >
> > > Hey everyone,
> > >
> > > TLDR: I would like a way of allowing network access during builds on a
> > > per-element basis, to ease transition into BuildStream.
> > >
> > > I realize this has a potential to be highly controversial, but I think that
> > > the benefits in terms of ease of adoption makes it worth it.
> >
> > In my opinion, this has the potential to cause more harm than good.
> > A good question is, if you are going to poke holes in the sandbox and
> > allow network access, what's the point of using BuildStream at all ?
>
> The point is about being able to incrementally improve the way your builds are
> made. It can be extremely hard, or too costly for a company to move everything
> to a 'correct' point before being able to move to BuildStream.
>
> This opens more an easy adoption of BuildStream, by allowing most of the
> benefits of it, and allowing to improve the 'bad' elements later, rather than
> having a way of solving every problem before moving to it.
>
> For me, this is a key point that BuildStream offered, compared to, for example,
> Bazel, which is more or less 'all or nothing'.
>
> Yes, having network access during your builds can be bad, and we should
> absolutely discourage that. However, I strongly believe that in order to help
> with adoption, users should be allowed to explicitely ask the permission to
> shoot themselves in the foot.
I'm not trying to block this outright, but still want to caution that
this really might do more harm than it does good.
Also, I would be more interested to see examples which are specifically
difficult to deal with, and work on making those use cases easier to
deal with.
This is after all why we developed plugins like the cargo plugin for
rust apps (some weird languages want to download dependencies from the
internet) - if we can write source plugins for these problematic
languages, this is a better avenue to pursue than to just give up and
open up the network at build time.
Do we have concrete examples of workflows which are difficult to port
which cannot be solved with source plugins we can write ?
[...]
> > If network access were "supported", this could lead us down a road
> > where we are requested to rebuild things when artifacts are too old
> > (for instance, I cannot build against this dependency, because the
> > random source code it downloaded from the internet only 3 months ago is
> > no longer valid for me to build against).
>
> Again, if you download from the internet that is a problem. If you have already
> a cache for such dependencies in your organization, that is not a concern.
I think the above is dangerous thinking, and there are certainly still
concerns.
Just because you are maintaining a store of sources in your own
organization does not really mean that those sources are not random,
and while it might work for a time (or even forever, if you are very
careful to always use it correctly), it will be difficult to maintain
and guarantee correctness of your builds over longer periods of time.
Essentially, this externalizes the problem of matching up a specific
set of inputs with a given build, making it the organizations problem
instead of using BuildStream to specifically address all inputs (with
checksums or git shas or the like)... I can see this being a cause of
frustration years down the line.
For instance, every few years when you fork your codebase for the next
generation model of your appliance, and you want to use a whole new set
of dependencies to build your new generation software against, then it
is up to you have separate caches for these deterministically stored
dependencies on your internal servers, it is fairly easy to screw up
and forget that you might have to land a hotfix for an older model that
is still supported in the field, and not be able to because you
"upgraded" you organizations dependency cache...
While this example applies over time, it also applies to building
products and appliances for different markets and clients, the
differing variants of your product may end up using differing versions
of the sources you host in an internal store, and you will need to
manually fork that internal store every time you fork the BuildStream
project(s) which builds your product.
This margin of error can be eliminated by ensuring that your
BuildStream project data addresses sources specifically and does not
tolerate variants (like tarball address without checksum for instance).
> > Mostly I worry that if we give users any opportunity to allow internet
> > access in builds, we almost guarantee that they will not do the minimal
> > legwork required to ensure their build works without network access.
>
> I believe that this is to the users to decide. We can make it very clear
> in the docs that this is a bad idea, and users can decide to shoot themselves
> in the foot.
>
[...]
> No, again, this is a documentation concern. We can document that by using this
> you understand your element is not repeatable etc.
>
> Fixing 'one element' is something that is doable indeed. Doing that at a
> company scale, on large projects become extremely hard. Being able to move to
> BuildStream before, allows to stop the bleeding for new elements and cleanup
> the rest on a longer term.
I am worried about underestimating the willingness of users to not fix
a problem when given any opportunity, I expect that given a simple
switch, people will always turn it on at any time any element gives
them trouble even if it was relatively easy to fix, and they will not
circle back to fix this problem element until it has hurt them, twice.
Regardless of whether we warned users in advance, they would be right
to blame us for not forcing them to do a small amount of work in
advance and save them from possible problems down the line.
In conclusion, these are my general feelings on the matter:
* If there are cases I've overlooked where a "source transform"
would not remedy the situation, then maybe we need a switch
like you suggest, but I'd really like to be enlightened as to
what those cases are.
* Right now we only have "source transforms" for python and rust
(the pip and the cargo sources).
This is a sad state of affairs, there are probably some other
weird languages doing similar things, node.js comes to mind.
Is it realistic for us to be able to cover every language which
downloads external dependencies from the internet at build time
with a source plugin (like pip/cargo) that we maintain in our
trusted plugin collection ?
I feel like this is realistic, but if not, then maybe it also
calls for such a switch.
* There are some cases which are ridiculously easy to fix, like
scripts which want to download config.sub and config.guess from
git.savannah.gnu.org (which sometimes need to be fixed to copy them
in from /usr/share/automake/ instead), or builds like cracklib
which might want to download it's word dictionary if the user
forgot to separately put it in place.
In these cases, I worry that providing a switch is going to
result in most of these easy to fix issues being completely
ignored and worked around, leading to a generally bad user
experience down the road.
Cheers,
-Tristan
Re: Conditionally open the network in the build sandbox
Posted by Benjamin Schubert <co...@benschubert.me>.
Hey,
Responses inline
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, October 14th, 2020 at 11:45 AM, Tristan Van Berkom <tr...@codethink.co.uk> wrote:
> Hi,
>
> On Fri, 2020-10-09 at 14:24 +0000,
>
> Benjamin Schubert wrote:
>
> > Hey everyone,
> >
> > TLDR: I would like a way of allowing network access during builds on a
> > per-element basis, to ease transition into BuildStream.
> >
> > I realize this has a potential to be highly controversial, but I think that
> > the benefits in terms of ease of adoption makes it worth it.
>
> In my opinion, this has the potential to cause more harm than good.
> A good question is, if you are going to poke holes in the sandbox and
> allow network access, what's the point of using BuildStream at all ?
The point is about being able to incrementally improve the way your builds are
made. It can be extremely hard, or too costly for a company to move everything
to a 'correct' point before being able to move to BuildStream.
This opens more an easy adoption of BuildStream, by allowing most of the
benefits of it, and allowing to improve the 'bad' elements later, rather than
having a way of solving every problem before moving to it.
For me, this is a key point that BuildStream offered, compared to, for example,
Bazel, which is more or less 'all or nothing'.
Yes, having network access during your builds can be bad, and we should
absolutely discourage that. However, I strongly believe that in order to help
with adoption, users should be allowed to explicitely ask the permission to
shoot themselves in the foot.
> The cache key loses a lot of meaning and no longer provides the kind of
> guarantees it should, that at least the source code being built is
> unique for the cache key...
This is a tangential problem. You can still have repeatable builds, with unique
cache keys even with network access. It is harder, and requires some
infrastructure, but it is doable, and many companies already have such
mechanism in place.
> If network access were "supported", this could lead us down a road
> where we are requested to rebuild things when artifacts are too old
> (for instance, I cannot build against this dependency, because the
> random source code it downloaded from the internet only 3 months ago is
> no longer valid for me to build against).
Again, if you download from the internet that is a problem. If you have already
a cache for such dependencies in your organization, that is not a concern.
> Mostly I worry that if we give users any opportunity to allow internet
> access in builds, we almost guarantee that they will not do the minimal
> legwork required to ensure their build works without network access.
I believe that this is to the users to decide. We can make it very clear
in the docs that this is a bad idea, and users can decide to shoot themselves
in the foot.
> Fixing your element to work without internet access can sometimes be a
> pesky and annoying task, and it is one which only pays off in the long
> run, which is where BuildStream provides your project real value. If
> people have an escape hatch to this, they will take it - and then when
> they suffer the consequences (only months or years later), they will be
> right to blame us for our false promises regarding repeatability.
No, again, this is a documentation concern. We can document that by using this
you understand your element is not repeatable etc.
Fixing 'one element' is something that is doable indeed. Doing that at a
company scale, on large projects become extremely hard. Being able to move to
BuildStream before, allows to stop the bleeding for new elements and cleanup
the rest on a longer term.
It also provides a much better visibility to which elements are not
(necessarily) repeatable, since you now have to state it explicitely.
So while users would be doing something that is not recommended, it would still
help them move towards better builds.
> I think this proposal is essentially an easy way to arm users against
> themselves, and I worry much more about the negative impacts this can
> have than I worry about the initial workload it takes to adopt a tool
> which forces you build your code repeatably and reliably.
This is a question of being able to migrate to a better integration build
system versus not being able to. If a huge amount of work is required before
moving, what do we offer over Bazel?
>
> Cheers,
>
> -Tristan
Thanks for your answer!
Cheers,
Ben
Re: Conditionally open the network in the build sandbox
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi,
On Fri, 2020-10-09 at 14:24 +0000, Benjamin Schubert wrote:
> Hey everyone,
>
> TLDR: I would like a way of allowing network access during builds on a
> per-element basis, to ease transition into BuildStream.
>
> I realize this has a potential to be highly controversial, but I think that
> the benefits in terms of ease of adoption makes it worth it.
>
In my opinion, this has the potential to cause more harm than good.
A good question is, if you are going to poke holes in the sandbox and
allow network access, what's the point of using BuildStream at all ?
The cache key loses a lot of meaning and no longer provides the kind of
guarantees it should, that at least the source code being built is
unique for the cache key...
If network access were "supported", this could lead us down a road
where we are requested to rebuild things when artifacts are too old
(for instance, I cannot build against this dependency, because the
random source code it downloaded from the internet only 3 months ago is
no longer valid for me to build against).
Mostly I worry that if we give users any opportunity to allow internet
access in builds, we almost guarantee that they will not do the minimal
legwork required to ensure their build works without network access.
Fixing your element to work without internet access can sometimes be a
pesky and annoying task, and it is one which only pays off in the long
run, which is where BuildStream provides your project real value. If
people have an escape hatch to this, they will take it - and then when
they suffer the consequences (only months or years later), they will be
right to blame us for our false promises regarding repeatability.
I think this proposal is essentially an easy way to arm users against
themselves, and I worry much more about the negative impacts this can
have than I worry about the initial workload it takes to adopt a tool
which forces you build your code repeatably and reliably.
Cheers,
-Tristan
> Problem
> -----------
> In some cases, it is very hard to completely sandbox a build, with
> some build systems absolutely wanting to go to the network. There are
> workaround to go over those, however, having to completely fix them before
> moving sizeable projects to BuildStream makes it very hard to adopt.
>
> In other cases, you might want access to network resources for testing elements
> that would require some network access
>
> I know and fully understand that having network access during a sandbox build
> removes many of the benefits of BuildStream (source caching, repeatable builds),
> but those can be addressed at an infrastructure level before being moved to
> work as it should in BuildStream.
>
> How would that look like?
> --------------------------
>
> I would envision a new variable option, similar to 'max-jobs', that would be
> transparent to the plugin, but used by BuildStream to open network access.
>
> How much work would we need to do for this?
> ------------------------------------------
>
> We already have mechanisms to open the sandbox when running a shell.
> We would need to make this possible for regular builds.
>
> This would also mean we need to be able to be able to set it on platform
> properties for remote execution.
>
> Mitigations
> -----------
>
> Some potential mitigations around this being 'bad' could be:
>
> - Optionally disable by default the pushing of such artifacts to remote caches,
> like we currently handle workspaces, with the ability to enable it either at
> the project level or the element level.
>
> Any thoughts?
>
> Thanks!
>
> Ben
Re: Conditionally open the network in the build sandbox
Posted by Jürg Billeter <j...@bitron.ch>.
Hi Ben,
On Fri, 2020-10-09 at 14:24 +0000, Benjamin Schubert wrote:
> TLDR: I would like a way of allowing network access during builds on a
> per-element basis, to ease transition into BuildStream.
>
> I realize this has a potential to be highly controversial, but I think that
> the benefits in terms of ease of adoption makes it worth it.
While it's obviously not ideal, I think it makes sense to support it if
there are real cases where this will significantly ease transition.
This would obviously need a big and clear disclaimer in the
documentation.
> [...]
>
> How would that look like?
> --------------------------
>
> I would envision a new variable option, similar to 'max-jobs', that would be
> transparent to the plugin, but used by BuildStream to open network access.
`max-jobs` is a configuration option, not an element variable. We could
follow the approach of the `notparallel` element variable. However, I
think the already existing `sandbox` section would be a better fit.
> How much work would we need to do for this?
> ------------------------------------------
>
> We already have mechanisms to open the sandbox when running a shell.
> We would need to make this possible for regular builds.
A potential issue is `resolv.conf`. Projects can bind-mount it in their
`shell` configuration. However, I don't know whether bind-mounting is a
good idea for build sandboxes. It would also leave the question of
where to configure it. And bind-mounting is not supported on non-
Linux/bubblewrap systems, although a fallback to copying the file may
be acceptable. Have you already thought about this?
> This would also mean we need to be able to be able to set it on platform
> properties for remote execution.
buildbox-run-bubblewrap already supports this with the platform
property `network` (used by `bst shell` for local execution), however,
this isn't a standard platform property.
> Mitigations
> -----------
>
> Some potential mitigations around this being 'bad' could be:
>
> - Optionally disable by default the pushing of such artifacts to remote caches,
> like we currently handle workspaces, with the ability to enable it either at
> the project level or the element level.
I don't think we need this and I'd rather keep it simple. The new
network setting must affect the element cache key but then I don't see
any real benefit in not pushing such artifacts.
It may make sense to print a warning message for every build job that
enables network access (which could be configured to be fatal, similar
to overlaps).
Another improvement would be to make the new configuration option a
list of host names to allow access to, instead of blanket network
access. A tiny DNS proxy could cover this at least to a certain degree
(without packet filtering or full IP proxy we can't prevent access by
IP address or alternative DNS server), or maybe even a generated
`hosts` file with entries for permitted hosts. This may also help solve
the `resolv.conf` issue.
Cheers,
Jürg