Proposal: Query Configuration from the bst client

[Tomaz Canabrava <tc...@kde.org>]

Hello,

I have only one configuration file for buildstream, on
~/.conf/buildstream.conf, and I configured as:

$ cat ~/.config/buildstream.conf | grep artifa
artifactdir: /data/Projects/Buildstream/build/artifactdir

but for some reason my buildstream assumed that it should store my
artifacts on ~/.cache/ filling up completely my disk and I had to create a
bootable usb stick using my wife's computer, as I could not even login
anymore (no space on /tmp, no space on /), /data my old 2tb HDD that I use
for compilation.

So I propose something simple, but that *might* solve this kind of issue, a
new addition to core (as I don't know how to provide this outside core, if
it's possible I'm all ears) that you can query all the maps and definitions
from the yaml configuration (or defaults, if none).

Example usage of the things I imagine:

$ bst query-config --config-file
/home/username/.conf/buildstream1.bst
/home/username/.conf/buildstream.bst

$ bst query-config --list-possible-keys
sourcedir: string      folder to store the sources
artifactdir: string      folder to store the cached artifacts
...
logdir: string            folder to store the logs

$ bst query-config --show-config
(result of print(yaml_map here))

$ bst query-config --key cachedir
cache-dir is only availble on buildstream 2.0, please check if you are
using the correct version

$ bst query-config --key "projects.gnome"
gnome: {
    strict: false
}

$ bst query-config --key "logdir"
/data/something/log

This way I could have queried the current file and folder used for
artifacts withotu assuming that the configuration file was correct - I
still don't know that it is.

Tougths?
Tomaz

Re: Proposal: Query Configuration from the bst client

[Tristan Van Berkom <tr...@codethink.co.uk>]

Hi Chandan,

On Mon, 2020-10-05 at 17:36 +0100, Chandan Singh wrote:
> Hi,
[...]
> > > In this case, I think it would be sensible to at least have an option to
> > > display the same pipeline heading (to stderr, which is considered UI, not
> > > output) in `bst show`, or to just display the heading to stderr
> > > unconditionally there... I think this should essentially satisfy your
> > > expressed use case without adding much or any API surface.
> > > 
> > > Thoughts ?
> > > 
> > That fixes my issues, yes.
> > something like `bst show --heading`
> 
> I'd be +1 for some command to query system info like cache usage etc.
> If such a flag was added as-is, note that it'd only work in the
> context of a BuildStream project. Some comments on that behavior
> follow.
> 
> Currently all `bst` CLI operations run with the `app.initialized()`
> context, and one of the first things it does is to make sure that we
> have a project to work with. So, if we want this to work even outside
> the context of a project, we might need to think about offering some
> CLI commands that don't require an initialized context. Maybe
> something like `bst system info`. This would leave the door open for
> other `system` commands like cache pruning etc for future, if we want
> to add those.

To me, this sounds like a perfectly sane separate conversation to have.

I am personally very keen on actually reaching 2.0, and I think we can
agree that such a command group you suggest, while certainly
interesting, need not block 2.0, which is why I think it is crucial at
this point to have granular conversations which we can conclude more
easily, and reach 2.0.

I have personally thought for a long time that the heading information
printed to stderr at `bst show` time would be both helpful and
harmless. It does show a lot of useful information about the loaded
pipeline such as the project options chosen to resolve the element
information output below, all this information deserves more attention
and prominence.

  * Can we agree on including the heading information in `bst show`
    output unconditionally ?

    This is my preference.

  * Can we agree on adding a `--print-heading/--no-print-heading`
    option to `bst show`, and add a user configuration so that the
    user can decide their preference ?

    I'm much less keen on this, as it would require another
    conversation about where to put user preference options in
    the config file which pertain to how the user prefers to see
    `bst show` output.

    I think that we can safely add the heading unconditionally now,
    and it would not be a breaking change to add such configuration
    in the future at some point if it became a strong desire.

Cheers,
    -Tristan

Re: Proposal: Query Configuration from the bst client

[Chandan Singh <ch...@chandansingh.net>]

Hi,

<snip>

> > > So I propose something simple, but that *might* solve this kind of
> > issue, a
> > > new addition to core (as I don't know how to provide this outside core,
> > if
> > > it's possible I'm all ears) that you can query all the maps and
> > definitions
> > > from the yaml configuration (or defaults, if none).
> >
> > I’m not fond of this idea honestly, I think we can get similar without
> > adding a whole new command.
> >
> > First off, every build log starts out by printing all such relevant
> > information in the heading, starting with BuildStream version, active user
> > configurations, loaded project configurations, and then summary of cache
> > keys for the loaded pipeline.
> >
> > This is printed every build.
> >
> > > Example usage of the things I imagine:
> > >
> > > $ bst query-config --config-file
> > > /home/username/.conf/buildstream1.bst
> > > /home/username/.conf/buildstream.bst
> >
> > What I mostly dislike about this, is that it would appear to offer some
> > kind of interoperability, require API stability, for wrapper scripts to
> > consume and presumably do something with the output, but we don’t offer any
> > kind of external API guarantees for things like the format of the internal
> > artifact cache or source cache directories.
> >
> > That said, judging from your proposal, the aim is to solve a different
> > problem, IIUC you would like a way to quickly inform the user.
> >
> > In this case, I think it would be sensible to at least have an option to
> > display the same pipeline heading (to stderr, which is considered UI, not
> > output) in `bst show`, or to just display the heading to stderr
> > unconditionally there... I think this should essentially satisfy your
> > expressed use case without adding much or any API surface.
> >
> > Thoughts ?
> >
> That fixes my issues, yes.
> something like `bst show --heading`

I'd be +1 for some command to query system info like cache usage etc.
If such a flag was added as-is, note that it'd only work in the
context of a BuildStream project. Some comments on that behavior
follow.

Currently all `bst` CLI operations run with the `app.initialized()`
context, and one of the first things it does is to make sure that we
have a project to work with. So, if we want this to work even outside
the context of a project, we might need to think about offering some
CLI commands that don't require an initialized context. Maybe
something like `bst system info`. This would leave the door open for
other `system` commands like cache pruning etc for future, if we want
to add those.

- Chandan

Re: Proposal: Query Configuration from the bst client

[Tomaz Canabrava <tc...@kde.org>]

On Mon, Oct 5, 2020 at 3:12 PM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:

> Hi Tomaz,
>
> > On Oct 4, 2020, at 11:41 PM, Tomaz Canabrava <tc...@kde.org> wrote:
> >
> > Hello,
> >
> > I have only one configuration file for buildstream, on
> > ~/.conf/buildstream.conf, and I configured as:
> >
> > $ cat ~/.config/buildstream.conf | grep artifa
> > artifactdir: /data/Projects/Buildstream/build/artifactdir
> >
> > but for some reason my buildstream assumed that it should store my
> > artifacts on ~/.cache/ filling up completely my disk and I had to create
> a
> > bootable usb stick using my wife's computer, as I could not even login
> > anymore (no space on /tmp, no space on /), /data my old 2tb HDD that I
> use
> > for compilation.
>
> My sympathies, this was an unfortunate experience to have encountered
> indeed.
>
> > So I propose something simple, but that *might* solve this kind of
> issue, a
> > new addition to core (as I don't know how to provide this outside core,
> if
> > it's possible I'm all ears) that you can query all the maps and
> definitions
> > from the yaml configuration (or defaults, if none).
>
> I’m not fond of this idea honestly, I think we can get similar without
> adding a whole new command.
>
> First off, every build log starts out by printing all such relevant
> information in the heading, starting with BuildStream version, active user
> configurations, loaded project configurations, and then summary of cache
> keys for the loaded pipeline.
>
> This is printed every build.
>
> > Example usage of the things I imagine:
> >
> > $ bst query-config --config-file
> > /home/username/.conf/buildstream1.bst
> > /home/username/.conf/buildstream.bst
>
> What I mostly dislike about this, is that it would appear to offer some
> kind of interoperability, require API stability, for wrapper scripts to
> consume and presumably do something with the output, but we don’t offer any
> kind of external API guarantees for things like the format of the internal
> artifact cache or source cache directories.
>
> That said, judging from your proposal, the aim is to solve a different
> problem, IIUC you would like a way to quickly inform the user.
>
> In this case, I think it would be sensible to at least have an option to
> display the same pipeline heading (to stderr, which is considered UI, not
> output) in `bst show`, or to just display the heading to stderr
> unconditionally there... I think this should essentially satisfy your
> expressed use case without adding much or any API surface.
>
> Thoughts ?
>


That fixes my issues, yes.
something like `bst show --heading`

Re: Proposal: Query Configuration from the bst client

[Tristan Van Berkom <tr...@codethink.co.uk>]

Hi Tomaz,

> On Oct 4, 2020, at 11:41 PM, Tomaz Canabrava <tc...@kde.org> wrote:
> 
> Hello,
> 
> I have only one configuration file for buildstream, on
> ~/.conf/buildstream.conf, and I configured as:
> 
> $ cat ~/.config/buildstream.conf | grep artifa
> artifactdir: /data/Projects/Buildstream/build/artifactdir
> 
> but for some reason my buildstream assumed that it should store my
> artifacts on ~/.cache/ filling up completely my disk and I had to create a
> bootable usb stick using my wife's computer, as I could not even login
> anymore (no space on /tmp, no space on /), /data my old 2tb HDD that I use
> for compilation.

My sympathies, this was an unfortunate experience to have encountered indeed.

> So I propose something simple, but that *might* solve this kind of issue, a
> new addition to core (as I don't know how to provide this outside core, if
> it's possible I'm all ears) that you can query all the maps and definitions
> from the yaml configuration (or defaults, if none).

I’m not fond of this idea honestly, I think we can get similar without adding a whole new command.

First off, every build log starts out by printing all such relevant information in the heading, starting with BuildStream version, active user configurations, loaded project configurations, and then summary of cache keys for the loaded pipeline.

This is printed every build.

> Example usage of the things I imagine:
> 
> $ bst query-config --config-file
> /home/username/.conf/buildstream1.bst
> /home/username/.conf/buildstream.bst

What I mostly dislike about this, is that it would appear to offer some kind of interoperability, require API stability, for wrapper scripts to consume and presumably do something with the output, but we don’t offer any kind of external API guarantees for things like the format of the internal artifact cache or source cache directories.

That said, judging from your proposal, the aim is to solve a different problem, IIUC you would like a way to quickly inform the user.

In this case, I think it would be sensible to at least have an option to display the same pipeline heading (to stderr, which is considered UI, not output) in `bst show`, or to just display the heading to stderr unconditionally there... I think this should essentially satisfy your expressed use case without adding much or any API surface.

Thoughts ?

Cheers,
    -Tristan


> $ bst query-config --list-possible-keys
> sourcedir: string      folder to store the sources
> artifactdir: string      folder to store the cached artifacts
> ...
> logdir: string            folder to store the logs
> 
> $ bst query-config --show-config
> (result of print(yaml_map here))
> 
> $ bst query-config --key cachedir
> cache-dir is only availble on buildstream 2.0, please check if you are
> using the correct version
> 
> $ bst query-config --key "projects.gnome"
> gnome: {
>    strict: false
> }
> 
> $ bst query-config --key "logdir"
> /data/something/log
> 
> This way I could have queried the current file and folder used for
> artifacts withotu assuming that the configuration file was correct - I
> still don't know that it is.
> 
> Tougths?
> Tomaz