You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Tristan Van Berkom <tr...@codethink.co.uk> on 2022/01/09 01:48:12 UTC
Updated 2.0 milestone (and wednesday January 5th meeting minutes)
Hi all,
Yesterday we finally got together to discuss BuildStream 2.0 again and
we made great strides towards putting this long overdue release on the
fast track.
The new milestone overview, which dictates what we expect to land before
our first candidate release, can be viewed here:
https://github.com/apache/buildstream/milestone/2
I've included the minutes below for reference.
Cheers,
-Tristan
Meeting minutes
===============
* Walk back over zealous plan to disallow writing in the sandbox
In our last meeting I had pushed for us to disallow plugins writing
to the sandbox using python code, as there have been some instances
of non-deterministic output as a result.
This turns out to be complicated and is dragging the release, so instead
just consider plugins who generate non-deterministic to be buggy and
need fixing.
VOTE: +1 unanimous
* Remove #1274 and #1275 from the blocker list
We had intended to use the remote asset cache to improve performance
of fetching and tracking when infrastructure allows for this.
Motion to not block on these features.
VOTE: +1 unanimous
NOTE: When implementing this after freezing the API, source plugins
will need to explicitly opt-in to participate in such
optimizations.
* Minimum time to market for plugin testing API
Discussion reached the following unanimous agreements:
- The 'buildstream.testing' API will be made private until it
can be deemed stable, as this need not block a 2.0 release.
- Some plugins which we intend to remove from BuildStream core
are required in order to preserve adequate test coverage of
the BuildStream core, in order to test the core API surfaces.
These plugins will be copied verbatim under the tests/ directory
before removing the plugins from BuildStream core, in order to
preserve test coverage.
NOTE: The ideal is still to produce minimal dummy plugins for testing
purposes, but for now there is no need to block on this.
* Plugin migration
Discussion reached the following unanimous agreements:
- Prominant plugins migrated outside of the BuildStream core for which we
intend to maintain, shall be migrated to a 'buildstream-plugins' repository
in apache github.
These shall be distributed on PyPI as a single package, even if the package
might not contain all plugins on platforms which lack support for potentially
platform specific plugins.
- These plugins include:
Source plugins: zip, git, bzr, pip, docker, cargo, patch
Element plugins: cmake, meson, autotools, make, pip, distutils
NOTE: The distutils element plugin will be renamed 'setuptools'
NOTE: The 'patch' source plugin may be promoted back to the core in the
case that we find that we need to use 'patch' with 'tar' in order
to load external plugins via junctions, but for now this is not
expected.
- Adding core plugins in the future could potentially cause namespace issues.
To solve this, we shall ensure that core plugin names are shadowed by
explicitly loaded external plugins
ACTION: Sander to work on obtaining our new 'buildstream-plugins' repo
ACTION: Tristan to file an issue regarding plugin name shadowing and add
this to the BuildStream 2.0 milestone (blocker list)
* Review remaining blockers on the milestone
Discussion reached the following unanimous agreements:
- Remove #1327 from the blocker list.
This was a requirement for a proposal regarding composite plugins, which
is complex and most probably unneeded - in the worst case it may result
in a less than perfect API to create the composite plugin mechanism.
- Remove #1385 from the blocker list.
This is the policy about cache keys being stable indefinitely.
At this stage there are no action items remaining to make this a reality,
and we need follow our policy of not breaking cache key stability in the
core.
NOTE: External plugins can still potentially break cache keys, or they
can be written in such a was as to preserve cache keys. The new
'junction' plugin origin which allows strict plugin version pinning
helps the consumer to avoid consuming plugins which break cache keys.
- Remove #1296 from the blocker list.
This issue proposes a nice feature to attempt to automatically upgrade
your project to be BuildStream 2.0 ready when running `bst init`, while
this is a nice feature, there is no need to block a release on this.
* Relicensing effort
After a long and arduous process of contacting contributors which continue
to have code changes that exist in the master branch, we are finally ready
to merge #1464 and formally relicense BuildStream under the apache license.
VOTE: +1 unanimous
Running build shells on non-build-elements [Was: Updated 2.0 milestone]
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi again Abderrahim,
On Mon, 2022-01-10 at 18:20 +0100, Abderrahim Kitouni wrote:
> Hi Tristan,
>
> Le lun. 10 janv. 2022 à 06:23, Tristan Van Berkom
> <tr...@codethink.co.uk> a écrit :
> > > I'd add to the list
> > > https://github.com/apache/buildstream/issues/1263
> > > I think this is potentially (Caching buildtrees only works with BuildElement)
> >
> > This, I think what we're really talking about here is script elements,
> > or similar concoctions, and we probably don't want to cache anything
> > for a variety of elements (like compose or filter elements).
>
> As is probably clear from my message, I pressed send without finishing
> my sentence.
>
> The issue isn't really about caching buildtrees, it's about running a
> shell in a failed sandbox. As such elements that don't have a sandbox
> (BST_RUN_COMMANDS = False) like compose or filter are not concerned by
> this at all. Indeed, this is mainly about the script element and
> possibly other custom elements (whether implemented using
> ScriptElement or not).
>
> The issue is currently that this core functionality that works fine
> with bst 1 simply doesn't work with anything that isn't a
> BuildElement, which would be a regression. I'm not advocating for
> supporting build trees for more elements, but rather to have an
> alternative for elements that don't have a build tree to still be able
> to shell into a failed sandbox, especially for local builds.
So what is unclear to me, is how serious exactly this regression is and
what important use case is not covered by what I proposed, which I
consider to be a pretty straight forward approach to solving the issue
(and doesn't require any real changes afaics except for a better error
message).
For instance, it's clearly not important to run a build shell on a
stack, filter or compose element.
For a script element, we *could* set `%{build-root}` to "/", but that
is probably better off being a choice which a user can make, since
doing so implies caching a potentially huge build tree (probably a good
choice though when configured to only cache failed builds).
The discussion on the issue at:
https://github.com/apache/buildstream/issues/1263#issuecomment-773013851
...suggests a complicated approach which requires some careful
recording and consideration of "marked" directories in the sandbox, and
what purpose they are marked for, which strikes me both as overkill and
more fragile than requiring plugins to store working data in the
`%{build-root}`.
On regressions in general... we have to consider that bst-2 is a
different beast with a different feature set.
Some features just aren't there, we no longer support `bst build --
track` or tracking in conjunction with any other command.
Also it's possible for the user to decide to never cache build trees
(even failed ones), in which case I think it's perfectly sane to just
say that caching of failed builds *must* be enabled if you want to use
build shells at all.
Cheers,
-Tristan
Re: Updated 2.0 milestone (and wednesday January 5th meeting minutes)
Posted by Abderrahim Kitouni <a....@gmail.com>.
Hi Tristan,
Le lun. 10 janv. 2022 à 06:23, Tristan Van Berkom
<tr...@codethink.co.uk> a écrit :
> > I'd add to the list
> > https://github.com/apache/buildstream/issues/1263
> > I think this is potentially (Caching buildtrees only works with BuildElement)
>
> This, I think what we're really talking about here is script elements,
> or similar concoctions, and we probably don't want to cache anything
> for a variety of elements (like compose or filter elements).
As is probably clear from my message, I pressed send without finishing
my sentence.
The issue isn't really about caching buildtrees, it's about running a
shell in a failed sandbox. As such elements that don't have a sandbox
(BST_RUN_COMMANDS = False) like compose or filter are not concerned by
this at all. Indeed, this is mainly about the script element and
possibly other custom elements (whether implemented using
ScriptElement or not).
The issue is currently that this core functionality that works fine
with bst 1 simply doesn't work with anything that isn't a
BuildElement, which would be a regression. I'm not advocating for
supporting build trees for more elements, but rather to have an
alternative for elements that don't have a build tree to still be able
to shell into a failed sandbox, especially for local builds.
Cheers,
Abderrahim
Re: Updated 2.0 milestone (and wednesday January 5th meeting minutes)
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Abderrahim,
On Sun, 2022-01-09 at 19:06 +0100, Abderrahim Kitouni wrote:
> Hi all,
>
> Allow me to start my reply with a rant. I don't like how the project
> is run right now. I think it would be better to invite non-core
> contributors and prominent users of buildstream to such meetings. I
> understand that it might not be possible if such meetings are held in
> person, but it would be nice to at least announce the meeting
> beforehand and allow them to weigh in (even if they're not allowed to
> vote).
Well, fair enough.
From my perspective, I have very little time available to get others
together so I'm very satisfied that after many months I was able to
gather 5 people in the same meeting room for 2 hours, and make some
executive decisions which help us to get this show back on the road.
> I'll add my comments inline.
>
> Le dim. 9 janv. 2022 à 02:48, Tristan Van Berkom
> <tr...@codethink.co.uk> a écrit :
> > The new milestone overview, which dictates what we expect to land before
> > our first candidate release, can be viewed here:
> >
> > https://github.com/apache/buildstream/milestone/2
>
> I think buildstream has a lot of rough edges right now to even
> consider a release candidate. I would consider this what needs to be
> done before a beta release, and then we'll need some serious bug
> fixing before a release candidate.
Sure, a beta release is a better word to use than a release candidate.
The *really* important point though is that we reach the hard API
freeze, which is a point at which we can actually tell users that
BuildStream 2 (beta or otherwise) is safe to use.
Once we reach that point, I think we can expect the project to get some
momentum again.
That said, thanks for collecting and sharing these links !
> I'd add to the list
> https://github.com/apache/buildstream/issues/1263
> I think this is potentially (Caching buildtrees only works with BuildElement)
This, I think what we're really talking about here is script elements,
or similar concoctions, and we probably don't want to cache anything
for a variety of elements (like compose or filter elements).
I think this can be easily solved with some policy though:
a.) If you think your element should have a build tree that gets
cached, then structure your element such that what you want
to be cached is under `%{build-root}` (of course the element
implementation can also *set* `%{build-root}`)
b.) Don't have the expectation that all elements under the sun
will cache their build trees, instead exit more gracefully
with an informative error explaining that the element in question
does not provide support for this.
I'll comment back on the issue, not sure that I'd want to block a beta
on this but blocking a real release could make sense. It also makes
sense to have separate beta/release milestones in github so we can
better track this stuff.
> This should definitely be considered as it would affect cache keys:
> https://github.com/apache/buildstream/issues/1270
>
> There is also this that we want to either accept or close as wontfix.
> https://github.com/apache/buildstream/issues/1384
Added these two to the milestone.
> Then there is a long list of bugs that need to be fixed (a lot of them
> are about non-strict mode). These would be fine to leave after the
> beta.
> https://github.com/apache/buildstream/issues/1304
> https://github.com/apache/buildstream/issues/1352
> https://github.com/apache/buildstream/issues/1360
> https://github.com/apache/buildstream/issues/1394
> https://github.com/apache/buildstream/issues/1455
> https://github.com/apache/buildstream/issues/1460
> https://github.com/apache/buildstream/issues/1461
> https://github.com/apache/buildstream/issues/1465
>
Yeah a lot of buggy behavior probably needs to be tracked in a second
milestone.
>
> > Meeting minutes
> > ===============
> >
> > * Walk back over zealous plan to disallow writing in the sandbox
>
> I think this is nice, it allows for some really nice optimizations. It
> should probably have an explicit API and a big fat warning about it
> though.
Agreed, that needs to be thought out when reviewing the Directory API:
https://github.com/apache/buildstream/issues/1294
Thanks a lot for the feedback !
Cheers,
-Tristan
Re: Updated 2.0 milestone (and wednesday January 5th meeting minutes)
Posted by Abderrahim Kitouni <a....@gmail.com>.
Hi all,
Allow me to start my reply with a rant. I don't like how the project
is run right now. I think it would be better to invite non-core
contributors and prominent users of buildstream to such meetings. I
understand that it might not be possible if such meetings are held in
person, but it would be nice to at least announce the meeting
beforehand and allow them to weigh in (even if they're not allowed to
vote).
I'll add my comments inline.
Le dim. 9 janv. 2022 à 02:48, Tristan Van Berkom
<tr...@codethink.co.uk> a écrit :
> The new milestone overview, which dictates what we expect to land before
> our first candidate release, can be viewed here:
>
> https://github.com/apache/buildstream/milestone/2
I think buildstream has a lot of rough edges right now to even
consider a release candidate. I would consider this what needs to be
done before a beta release, and then we'll need some serious bug
fixing before a release candidate.
I'd add to the list
https://github.com/apache/buildstream/issues/1263
I think this is potentially (Caching buildtrees only works with BuildElement)
This should definitely be considered as it would affect cache keys:
https://github.com/apache/buildstream/issues/1270
There is also this that we want to either accept or close as wontfix.
https://github.com/apache/buildstream/issues/1384
Then there is a long list of bugs that need to be fixed (a lot of them
are about non-strict mode). These would be fine to leave after the
beta.
https://github.com/apache/buildstream/issues/1304
https://github.com/apache/buildstream/issues/1352
https://github.com/apache/buildstream/issues/1360
https://github.com/apache/buildstream/issues/1394
https://github.com/apache/buildstream/issues/1455
https://github.com/apache/buildstream/issues/1460
https://github.com/apache/buildstream/issues/1461
https://github.com/apache/buildstream/issues/1465
> Meeting minutes
> ===============
>
> * Walk back over zealous plan to disallow writing in the sandbox
I think this is nice, it allows for some really nice optimizations. It
should probably have an explicit API and a big fat warning about it
though.
> [...]
All the others seem fine.
Regards,
Abderrahim