You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by GitBox <gi...@apache.org> on 2022/10/07 16:16:07 UTC
[GitHub] [buildstream] staehle opened a new issue, #1773: Feature: Artifact Checkout Default Behavior
staehle opened a new issue, #1773:
URL: https://github.com/apache/buildstream/issues/1773
Hi, me again. (If there's a better way for me to submit things like this, let me know. There isn't really a clear "how to contact BuildStream Dev team")
When doing a "bst artifact checkout", the default argument to "--deps" is "run". Can this be made a user-configurable option so we can set it to "none" by default via https://docs.buildstream.build/master/using_config.html or as a project.conf value?
Reasoning:
The components we're cross-compiling have these base dependencies as "build and runtime" dependencies:
```
depends:
- base/base.bst
- toolchain/cross-compiler.bst
```
When running "bst artifact checkout", the artifact list is as I would expect (using cross-compiled zlib as an example):
```
$ bst -o target_arch aarch64 -o platform rpi artifact list-contents components/target/zlib.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
components/target/zlib.bst:
robs
robs/staging
robs/staging/image
robs/staging/image/usr
robs/staging/image/usr/lib
robs/staging/image/usr/lib/libz.so
robs/staging/image/usr/lib/libz.so.1
robs/staging/image/usr/lib/libz.so.1.2.12
robs/staging/sysroot
robs/staging/sysroot/usr
robs/staging/sysroot/usr/include
robs/staging/sysroot/usr/include/zconf.h
robs/staging/sysroot/usr/include/zlib.h
robs/staging/sysroot/usr/lib
robs/staging/sysroot/usr/lib/libz.a
robs/staging/sysroot/usr/lib/libz.so
robs/staging/sysroot/usr/lib/libz.so.1
robs/staging/sysroot/usr/lib/libz.so.1.2.12
robs/staging/sysroot/usr/share
robs/staging/sysroot/usr/share/man
robs/staging/sysroot/usr/share/man/man3
robs/staging/sysroot/usr/share/man/man3/zlib.3
robs/staging/sysroot/usr/share/pkgconfig
robs/staging/sysroot/usr/share/pkgconfig/zlib.pc
```
However, this is inconsistent with what "bst artifact checkout" does, which outputs the artifacts as well as the entire sandbox (...why?). I would only be expecting the "robs" directory inside "components/target/zlib/":
```
$ bst -o target_arch aarch64 -o platform rpi artifact checkout components/target/zlib.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][bd873de7][ main:components/target/zlib.bst ] START Staging dependencies
[00:00:00][bd873de7][ main:components/target/zlib.bst ] SUCCESS Staging dependencies
[--:--:--][bd873de7][ main:components/target/zlib.bst ] START Integrating sandbox
[00:00:00][bd873de7][ main:components/target/zlib.bst ] SUCCESS Integrating sandbox
[--:--:--][bd873de7][ main:components/target/zlib.bst ] START Checking out files in '<redacted>/components/target/zlib'
[00:00:01][bd873de7][ main:components/target/zlib.bst ] SUCCESS Checking out files in '<redacted>/components/target/zlib'
$ ls components/target/zlib/
bin dev etc lib lib64 proc robs sbin tmp usr
```
So we initially thought this was an issue with how we were handling dependencies, so had changed the base 'depends' to 'build-depends':
```
build-depends:
- base/base.bst
- toolchain/cross-compiler.bst
```
And this fixed "artifact checkout" to be what we'd expect. However, it broke "bst shell" since then the base sandbox wasn't there:
```
$ bst -o target_arch aarch64 -o platform rpi shell components/target/zlib.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Initializing remote caches
[00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches
[--:--:--][ ][ main:core activity ] START Query cache
[00:00:00][ ][ main:core activity ] SUCCESS Query cache
[--:--:--][bd873de7][ main:components/target/zlib.bst ] START Staging dependencies
[00:00:00][bd873de7][ main:components/target/zlib.bst ] SUCCESS Staging dependencies
[--:--:--][bd873de7][ main:components/target/zlib.bst ] START Integrating sandbox
[00:00:00][bd873de7][ main:components/target/zlib.bst ] SUCCESS Integrating sandbox
[--:--:--][bd873de7][ main:components/target/zlib.bst ] STATUS Running command
sh -i
Error launching shell: Staged artifacts do not provide command 'sh'
```
It was only **_after_** all this that I finally noticed that "artifact checkout" has a "--deps" option, which is "run" by default: https://docs.buildstream.build/master/using_commands.html#cmdoption-bst-artifact-checkout-d
Which I'm sure there's a reason for, but the inconsistency between "checkout" and "list-contents" was confusing for multiple people.
So, being able to change this default value in some config file would be nice, since our workflow would have us only ever really using that style. Also, the command line is getting quite long with the options we need (related #1770). For example: `$ bst -o target_arch aarch64 -o platform rpi -o debug true artifact checkout --deps none components/target/zlib.bst`, which is kinda... yeah.
Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] staehle commented on issue #1773: Feature: Artifact Checkout Default Behavior
Posted by GitBox <gi...@apache.org>.
staehle commented on issue #1773:
URL: https://github.com/apache/buildstream/issues/1773#issuecomment-1271870284
Thinking more about this... Changing the base sandbox to be in "build-depends" makes more sense, because it's not really logical to claim that the x86->arm cross-compiler is a _runtime_ dependency of an aarch64 zlib.so
I could pass `--build` to `bst shell`, but that argument actually does more than what your docs state. Without it by default, it stages everything as if you're looking at the sandbox after `bst build` completes, so the artifact results of that build are also placed in the final locations. And with `--build`, it's like it's just before `bst build` runs commands.
So if the sandbox is in the "build-depends", what is the intended use-case for `bst shell` without the `--build` option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] cs-shadow commented on issue #1773: Feature: Artifact Checkout Default Behavior
Posted by "cs-shadow (via GitHub)" <gi...@apache.org>.
cs-shadow commented on issue #1773:
URL: https://github.com/apache/buildstream/issues/1773#issuecomment-1688914893
Hello @staehle , Apologies for the late reply but I thought better late than never!
> Thinking more about this... Changing the base sandbox to be in "build-depends" makes more sense, because it's not really logical to claim that the x86->arm cross-compiler is a runtime dependency of an aarch64 zlib.so
I concur. The build sandbox, and build-related "stuff" in general, are best placed as build-time-only dependencies (aka `build-depends`).
> I could pass --build to bst shell, but that argument actually does more than what your docs state. Without it by default, it stages everything as if you're looking at the sandbox after bst build completes, so the artifact results of that build are also placed in the final locations. And with --build, it's like it's just before bst build runs commands.
>
> So if the sandbox is in the "build-depends", what is the intended use-case for bst shell without the --build option?
I can provide some context here. The `bst shell` command has 2 rather distinct use-cases - with and without `--build`:
* with `--build` - sometimes called "build shell", this mode is intended to help with debugging build failures. Let's say your build was failing and you wanted to play with some build flags, the build shell allows you to do so in the exact same environment (more or less) as `bst build` would do. As such, it provides all the build dependencies of the product and the sources. (note that runtime dependencies are not provided.)
* without `--build` - sometimes called "runtime shell" or just "shell", this mode is intended to debug the built artifacts together with their runtime dependencies. (note that sources are not mounted by default.)
With that disclaimer aside, my suggestion for your usecase would be to do one of the following:
1. Instead of running a shell in the sandbox, use the explicit command for testing like `bst shell TARGET -- COMMAND ARGS`. This may not always be possible but works well when testing a CLI of some sort.
2. Use some sort of base OS layer as a runtime dependency. This need not be the whole build sandbox, but at least enough things to be able to run a shell if you're after interactive debugging. It could potentailly be an option to have `bst shell` accept multiple targets. That would make it possible to do something like `bst shell TARGET RUNTIME-TARGET ANOTHER-TARGET -- COMMAND`, in case it's not desirable to permanently add a given runtime dependency.
----
In terms of consistency, both `shell` and `artifact checkout` commands default to `--deps run` as both stage the artifacts in some way. `artifact list-contents` is the outlier here, as it's sort of a plumbing command. Although I can totally see why that might be confusing.
----
Now Coming back to your original questions :)
> When doing a "bst artifact checkout", the default argument to "--deps" is "run". Can this be made a user-configurable option so we can set it to "none" by default via https://docs.buildstream.build/master/using_config.html or as a project.conf value?
Given this context, would you say this is still needed?
> Hi, me again. (If there's a better way for me to submit things like this, let me know. There isn't really a clear "how to contact BuildStream Dev team")
For new features, we generally recommend our mailing list as the first port of call. Some details here: https://docs.buildstream.build/master/CONTRIBUTING.html#adding-new-features
This is certainly the right place for any bug reports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org