You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Justin Erenkrantz <ju...@erenkrantz.com> on 2020/12/28 19:29:17 UTC

Re: Conditionally open the network in the build sandbox

On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
wrote:

> Exporting the build artifacts would certainly be one option. But
> depending on what you're trying to achieve, there may be other
> alternatives too. `bst shell` in particular comes to mind.
>
> The shell functionality already allows network access for purposes
> like this, and comes in two variants - build shell and runtime shell.
>
> The build shell will have all the build dependencies and sources for
> the element in exactly the same way that they would be staged during a
> `bst build`. It can be used to debug builds of the project, for
> example, to iterate on various configuration options.
>
> The runtime shell will provide access to the built artifacts  of the
> element and all its runtime dependencies. As such, it is more useful
> for testing the resulting binary.
>
> Both the shells allow network access. As such, they are not meant to
> produce deployable artifacts, but can be very useful for debugging
> purposes. There are probably still many things that could be improved
> in the shell experience. So, I'd be very interested for you to try the
> shell functionality and let us know if that works for your use case.
>
> One issue that I can already think of is that if we go with this
> approach, you will also need to have a debugger available in the
> BuildStream project and somehow add a dependency on it  for invoking
> the build shell. This topic has been a recurring discussion item, but
> we haven't solved it yet.
> https://gitlab.com/BuildStream/buildstream/-/issues/422 has some
> related background on this.
>
> A couple of references below.
>
> CLI docs:
> https://docs.buildstream.build/master/using_commands.html#invoking-shell
> Shell in action:
>
> https://docs.buildstream.build/master/tutorial/running-commands.html#run-the-hello-world-program


Thanks.  I'm picking this up again as there's an upstream change in OpenSSL
which is apparently breaking serf.  For reference, my serf-bst 2.0 project
is at [1].

Buildstream is *so* close to doing what I want here: control my version of
OpenSSL and Serf in a clean manner and do so in an isolated manner from my
host environment.

However, I need the debugger available to step through the test binaries to
figure out what's going on.  It also seems like the bst shell doesn't have
network access - I can't run any of the produced binaries to remote hosts.
Is there a step I'm missing to allow network access?  And, is the only way
to bring in the debugger is to build it into the base platform (or have a
special layer with gdb and such)?

I'm going to put this on pause for now and go back to fighting Debian sid
to see if I can get a reproducer...

Cheers.  -- justin

1. https://github.com/jerenkrantz/serf-bst

Re: Conditionally open the network in the build sandbox

Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Justin,

On Thu, 2020-12-31 at 12:19 -0500, Justin Erenkrantz wrote:
> On Tue, Dec 29, 2020 at 5:00 AM Tristan Van Berkom <
> tristan.vanberkom@codethink.co.uk> wrote:
> 
[...]
> Yup, it was the lack of resolv.conf - thanks!  I'll incorporate some of
> these tricks into serf-bst's project.conf.  Does it make sense to have
> slightly saner defaults in buildstream?

Not really.

BuildStream itself is only a build tool and is as decoupled as possible
from the product being built with the tool.

As such, it is a goal for us to remove as much as we can from the
default project.conf settings before 2.0:

  https://github.com/apache/buildstream/blob/master/src/buildstream/data/projectconfig.yaml

As you can see, most of the settings already present you are building a
GNU/Linux system, or something that will run on such a system.

We are hoping to remove anything that is payload specific from such
settings, so that these can be built from the ground up for any given
project and any OS we might support.

That said, this does not preclude sharing of default settings across
projects, for instance one could share defaults using cross junction
includes:

  https://docs.buildstream.build/1.93.5/junctions/junction-includes.html

[...]
> 
> Well, the issue is that there is a vicious cycle between gdb's source
> dependencies.
> 

Cyclic dependencies are an unfortunate fact of life indeed :)

> I tried creating elements for gdb - but, it requires makeinfo to build due
> to this issue:
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=14678
> 
> Texinfo requires gettext which requires wget...which requires makeinfo to
> build.
> 
> So, I can't technically define an upstream element for gdb from source.
> So, it must be injected outside Buildstream (which we have via our platform
> Docker image)...unless there's another trick to break these dependencies
> somehow?

With your current approach, you would want a gdb that is built for the
runtime which you are using, in which case indeed it would be best to
take the gdb which corresponds to the docker image you are based on.

This could be done with an additional import/filter element to select
only the gdb bits from your "platform Docker image".

Alternatively, you might consider basing your project on
freedesktop-sdk instead of an opaque docker image, or try building it
in a similar way as freedesktop-sdk is doing it:

    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/elements/components/gdb.bst

Interestingly, we do not build makeinfo with freedesktop-sdk, but we
also build it directly from git rather than a dist tarball.


That said, no matter how you fry the cat you still run into the issue
#422 I mentioned earlier.

I.e. you will have to either have the end result actually depend on gdb
or not, until we add a way to stage multiple elements/targets into the
same sandbox at `bst shell` time.


> Related to the above, is the still unresolved issue #413:
> >   https://gitlab.com/BuildStream/buildstream/-/issues/413
> > 
> > This pertains to adding build directories to sandbox for `bst shell`,
> > at the same locations where they were originally built, so that
> > debuggers can easily find the sources for the binaries they were built
> > for.
> > 
> 
> Also, it doesn't look like we're keeping the debug symbols around - so,
> even if I inject gdb into the platform, I don't have the debug symbols.
> I'll hunt around to figure out if there's an obvious approach.  So, I'm not
> even getting to the point of having the symbols linking to the
> (non-existent) source files...
> 

In your case, it may just be how the toolchain is configured in the
docker image you are extracting your runtime from. Given that you are
using a Docker base, I would not expect that base to include debug
symbols for the dependencies, but that does not necessarily mean you
would not have them for the libraries/apps which you are building.


More details
~~~~~~~~~~~~
If "strip-binaries" is not defined at all (which is default), then
compiled binaries do not have their debug info forcefully stripped from
them.

With fdsdk, it seems that they have baked something pretty custom for
some reason or another, but essentially the base toolchain is
generating output with debug information included, and this debug
information gets stripped out of the binaries and installed separately,
via the "%{strip-binaries}" variable declaration:

  https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/include/strip.yml

The "freedesktop-sdk-stripper" program is built manually from the files
at:

  https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/tree/master/files/stripper

Normally this is done with a `find "%{install-root}" -exec ..` loop
which invokes `objdump` and puts the debug info into corresponding
files in "%{libdir}/debug".

> > Of course, when we start talking about graphical debuggers and IDEs
> > then this becomes a much more complex discussion.
> > 
> 
> One point Sander mentioned is that some of the IDEs support remote symbol
> serving - so, perhaps something we can explore that might ease the barrier
> to entry.
> 

Exactly, this one line of progress we discussed, which is still quite
complex, as the symbols are inside the sandbox and need to be served
from the sandbox in such a way that the IDE can discover it.

It is a lot of legwork to come up with a design which makes it (fairly)
easy enough for projects to setup and work with, I suppose the first
step would be to create a hacky prototype for discovery purposes and
then design around that.

Cheers,
    -Tristan



Re: Conditionally open the network in the build sandbox

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Tue, Dec 29, 2020 at 5:00 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:

> Hi Justin,
>
> On Mon, 2020-12-28 at 14:29 -0500, Justin Erenkrantz wrote:
> > On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
> > wrote:
> [...]
> > Thanks.  I'm picking this up again as there's an upstream change in
> OpenSSL
> > which is apparently breaking serf.  For reference, my serf-bst 2.0
> project
> > is at [1].
> >
> > Buildstream is *so* close to doing what I want here: control my version
> of
> > OpenSSL and Serf in a clean manner and do so in an isolated manner from
> my
> > host environment.
>
> I took a look at your linked serf-bst project, and I feel like I am
> missing some context to be properly equipped to help more.
>
> What I am seeing is a docker used to create an input runtime upon which
> you are building some software, but I don't understand the context
> under which this is intended to be deployed or run.
>

The intention is to create the binary artifacts associated with the
dependency tree in a controlled manner.

Serf itself is just an HTTP library intended to be consumed by other
projects (such as Subversion, OpenOffice, etc, etc).

While there isn't a lot of activity in the project itself, there has been
recent churn in OpenSSL which silently breaks their API contracts.  So,
having an integration build be able to easily swap out versions of OpenSSL
is very very useful.

> However, I need the debugger available to step through the test binaries
> to
> > figure out what's going on.  It also seems like the bst shell doesn't
> have
> > network access - I can't run any of the produced binaries to remote
> hosts.
> > Is there a step I'm missing to allow network access?  And, is the only
> way
> > to bring in the debugger is to build it into the base platform (or have a
> > special layer with gdb and such)?
>
> Regarding network access in `bst shell`
> ---------------------------------------
> It appears that network access should be enabled by default, and
> specifying the `--isolate` flag will ensure that there is no network
> access.
>
> That said, it is usually not enough to merely allow networking in order
> to have a networking enabled shell, to get networking to actually work,
> you will probably at least need to mount your host's /etc/resolv.conf
> into the shell environment.
>
> You can try this with:
>
>   bst shell --mount /etc/resolv.conf /etc/resolv.conf serf.bst
>
> It is recommended to setup your preferred shell configuration in your
> project.conf to make things easier to run the sorts of software you
> build in a shell for testing/debugging:
>
>   See the examples below for the `host-files` configuration:
>
> https://docs.buildstream.build/master/format_project.html#customizing-the-shell
>
> Here is a more elaborate example of a shell configuration which allows
> the user to launch graphics applications on (most) linux variants which
> use X (or XWayland) and pulseaudio:
>
>
> https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/project.conf#L165


Yup, it was the lack of resolv.conf - thanks!  I'll incorporate some of
these tricks into serf-bst's project.conf.  Does it make sense to have
slightly saner defaults in buildstream?

Regarding debugging tool presence in `bst shell`
> ------------------------------------------------
> This has been the subject of much debate a couple of years ago.
>
> In short: Yes, the only reliable way is to build your tools or wrap up
> some binaries known to work in that project's specific runtime
> environment and put them there, that means using elements.
>
> That however only answers half of the question when it comes to
> practical use of debuggers in `bst shell`, the other half is that a
> project does not truly depend on debuggers and profilers and such
> tooling.
>
> As far as I can see the conclusion of these debates is issue #422:
>
>   https://gitlab.com/BuildStream/buildstream/-/issues/422
>
> This states that basically one should be allowed to stage multiple
> elements into the same `bst shell` environment, possibly resulting in
> issuing some file overlap warnings.
>
> This should not be all that difficult to implement, and implementing
> this would mean that one could have a defined set of debugging tools
> which one could optionally stage in `bst shell`.
>

Well, the issue is that there is a vicious cycle between gdb's source
dependencies.

I tried creating elements for gdb - but, it requires makeinfo to build due
to this issue:

https://sourceware.org/bugzilla/show_bug.cgi?id=14678

Texinfo requires gettext which requires wget...which requires makeinfo to
build.

So, I can't technically define an upstream element for gdb from source.
So, it must be injected outside Buildstream (which we have via our platform
Docker image)...unless there's another trick to break these dependencies
somehow?

Related to the above, is the still unresolved issue #413:
>
>   https://gitlab.com/BuildStream/buildstream/-/issues/413
>
> This pertains to adding build directories to sandbox for `bst shell`,
> at the same locations where they were originally built, so that
> debuggers can easily find the sources for the binaries they were built
> for.
>

Also, it doesn't look like we're keeping the debug symbols around - so,
even if I inject gdb into the platform, I don't have the debug symbols.
I'll hunt around to figure out if there's an obvious approach.  So, I'm not
even getting to the point of having the symbols linking to the
(non-existent) source files...


> Of course, when we start talking about graphical debuggers and IDEs
> then this becomes a much more complex discussion.
>

One point Sander mentioned is that some of the IDEs support remote symbol
serving - so, perhaps something we can explore that might ease the barrier
to entry.

Cheers.  -- justin

Re: Conditionally open the network in the build sandbox

Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Justin,

On Mon, 2020-12-28 at 14:29 -0500, Justin Erenkrantz wrote:
> On Tue, Oct 27, 2020 at 3:28 PM Chandan Singh <ch...@chandansingh.net>
> wrote:
[...]
> Thanks.  I'm picking this up again as there's an upstream change in OpenSSL
> which is apparently breaking serf.  For reference, my serf-bst 2.0 project
> is at [1].
> 
> Buildstream is *so* close to doing what I want here: control my version of
> OpenSSL and Serf in a clean manner and do so in an isolated manner from my
> host environment.

I took a look at your linked serf-bst project, and I feel like I am
missing some context to be properly equipped to help more.

What I am seeing is a docker used to create an input runtime upon which
you are building some software, but I don't understand the context
under which this is intended to be deployed or run.

> However, I need the debugger available to step through the test binaries to
> figure out what's going on.  It also seems like the bst shell doesn't have
> network access - I can't run any of the produced binaries to remote hosts.
> Is there a step I'm missing to allow network access?  And, is the only way
> to bring in the debugger is to build it into the base platform (or have a
> special layer with gdb and such)?

Regarding network access in `bst shell`
---------------------------------------
It appears that network access should be enabled by default, and
specifying the `--isolate` flag will ensure that there is no network
access.

That said, it is usually not enough to merely allow networking in order
to have a networking enabled shell, to get networking to actually work,
you will probably at least need to mount your host's /etc/resolv.conf
into the shell environment.

You can try this with:

  bst shell --mount /etc/resolv.conf /etc/resolv.conf serf.bst

It is recommended to setup your preferred shell configuration in your
project.conf to make things easier to run the sorts of software you
build in a shell for testing/debugging:

  See the examples below for the `host-files` configuration:
  https://docs.buildstream.build/master/format_project.html#customizing-the-shell

Here is a more elaborate example of a shell configuration which allows
the user to launch graphics applications on (most) linux variants which
use X (or XWayland) and pulseaudio:

  https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/master/project.conf#L165


Regarding debugging tool presence in `bst shell`
------------------------------------------------
This has been the subject of much debate a couple of years ago.

In short: Yes, the only reliable way is to build your tools or wrap up
some binaries known to work in that project's specific runtime
environment and put them there, that means using elements.

That however only answers half of the question when it comes to
practical use of debuggers in `bst shell`, the other half is that a
project does not truly depend on debuggers and profilers and such
tooling.

As far as I can see the conclusion of these debates is issue #422:

  https://gitlab.com/BuildStream/buildstream/-/issues/422

This states that basically one should be allowed to stage multiple
elements into the same `bst shell` environment, possibly resulting in
issuing some file overlap warnings.

This should not be all that difficult to implement, and implementing
this would mean that one could have a defined set of debugging tools
which one could optionally stage in `bst shell`.

Related to the above, is the still unresolved issue #413:

  https://gitlab.com/BuildStream/buildstream/-/issues/413

This pertains to adding build directories to sandbox for `bst shell`,
at the same locations where they were originally built, so that
debuggers can easily find the sources for the binaries they were built
for.

Of course, when we start talking about graphical debuggers and IDEs
then this becomes a much more complex discussion.

Cheers,
    -Tristan