You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Martín Abente Lahaye <ma...@codethink.co.uk.INVALID> on 2024/11/21 10:42:23 UTC
Extending Sources public APIs
Hello everyone,
Currently, some of our community plugins like collect_manifest [1] and
tools like bst-to-lorry [2], rely on a combination of assumptions based
on the reported “kind” of the Source and private Python APIs.
This can be problematic as both collect_manifest and bst-to-lorry query
sources for their "kind" and assume that certain attributes and methods
will be present (e.g., source.url). In fact, this has been discussed
before at least once [3].
Although this seems to work, it’s unreliable because even if the “kind”
string matches, there’s no guarantee that it is the expected Source, as
it can be a different Source with the same “kind” string. Plus, even if
it really is the expected Source, a future refactor could break these
assumptions as these aren’t public APIs.
Ultimately, what both of these are trying to determine is:
* The full upstream URL to the source, e.g.,
https://sourceware.org/git/glibc.git, to include it in the output
manifest or the lorry configuration file.
* The version of each source, e.g., 2.40, to include it in the output
manifest.
* Other information such as what the source is configured to track, to
include it in the lorry configuration file.
Therefore, instead of letting plugins and tools do all that unreliable
guessing, we could provide what these ultimately need by adding new
abstract methods that each Source can implement. For example, something
like:
* Source.get_urls(self) -> List[str]: Which would provide a list of full
upstream URL without any guessing or relying on accessing private
attributes, for the caller.
* Source.get_versions(self) -> List[str]: Similarly, for the versions,
but the tricky piece with this would be the need for a regexp for each
source, e.g., in case the version needs to be extracted from the Source
URL.
* Source.get_trackings(self) -> List[Optional[str]]: Similarly, for the
tracking strings. Of course this would only make sense for sources that
can actually be tracked.
Or perhaps, something that better groups these tuples, e.g.,
Source.get_actual_sources(self) -> List[Tuple[str, str, Optional[str]]],
providing URL, version and tracking strings tuples or equivalent object.
A key question here is whether something like the above would still be
too rigid or over-specified for that plugin and tool, and perhaps we
should be thinking of a more free-form API to query for these.
An idea that was mentioned when discussing this topic with Abderrahim
was to introduce to sources something similar to the elements public
data, e.g., this way we could add that version matching regexp.
What do you all think?
Regards,
Martín.
Refs:
[1]
https://gitlab.com/BuildStream/buildstream-plugins-community/-/blob/22023ad60e91ff3f635c556ed4c32ce4dfd7c2b5/src/buildstream_plugins_community/elements/collect_manifest.py
[2]
https://gitlab.com/CodethinkLabs/lorry/bst-to-lorry/-/blob/d6d3782071502c56611ceffa574d2f81e2a1eedd/bst_to_lorry.py
[3]
https://gitlab.com/BuildStream/buildstream-plugins-community/-/issues/2
Re: Extending Sources public APIs
Posted by Abderrahim Kitouni <a....@gmail.com>.
Hello Martín,
Thanks for the write up. See my comments inline below.
Le jeu. 21 nov. 2024 à 11:42, Martín Abente Lahaye
<ma...@codethink.co.uk.invalid> a écrit :
> Ultimately, what both of these are trying to determine is:
>
> * The full upstream URL to the source, e.g.,
> https://sourceware.org/git/glibc.git, to include it in the output
> manifest or the lorry configuration file.
> * The version of each source, e.g., 2.40, to include it in the output
> manifest.
> * Other information such as what the source is configured to track, to
> include it in the lorry configuration file.
I think this is pretty accurate. I would just add that it tries to
guess the version based on the tarball name or the git describe output
for a git repository (which it assumes is the format used by the ref).
> Therefore, instead of letting plugins and tools do all that unreliable
> guessing, we could provide what these ultimately need by adding new
> abstract methods that each Source can implement. For example, something
> like:
>
> * Source.get_urls(self) -> List[str]: Which would provide a list of full
> upstream URL without any guessing or relying on accessing private
> attributes, for the caller.
> * Source.get_versions(self) -> List[str]: Similarly, for the versions,
> but the tricky piece with this would be the need for a regexp for each
> source, e.g., in case the version needs to be extracted from the Source
> URL.
> * Source.get_trackings(self) -> List[Optional[str]]: Similarly, for the
> tracking strings. Of course this would only make sense for sources that
> can actually be tracked.
I'd add as prior art what was done in buildstream-plugins-community
(for collect_manifest plugin, but was extended for bst-to-lorry). See
[1]
Source.export_manifest(self) -> Dict[str, str]
This returns a more or less free-form dictionary with keys defined in
the collect_manifest plugin for a few things
* "type" for the type of the source (archive/git/patch)
* "url" or "path" for the provenance
* "commit" or "sha256" for the ref
As it's freeform, it can be extended to support other things, such as
the tracked branch which is read by bst-to-lorry.
> Or perhaps, something that better groups these tuples, e.g.,
> Source.get_actual_sources(self) -> List[Tuple[str, str, Optional[str]]],
> providing URL, version and tracking strings tuples or equivalent object.
This reminds me of the proposal to use the Remote Asset API [2] for
fetching the sources. If we end up going with an idea similar to this,
it should at least take into consideration a future use of Remote
Asset.
> An idea that was mentioned when discussing this topic with Abderrahim
> was to introduce to sources something similar to the elements public
> data, e.g., this way we could add that version matching regexp.
This would essentially be another way to have sources export free-form
data to other plugins. It's interesting in that we would reproduce the
same concept that we already have for elements, rather than inventing
something new.
Cheers,
Abderrahim
[1] https://buildstream.gitlab.io/buildstream-plugins-community/elements/collect_manifest.html
[2] https://github.com/apache/buildstream/issues/1274
Re: Extending Sources public APIs
Posted by Abderrahim Kitouni <a....@gmail.com>.
Hi Harry,
Le jeu. 21 nov. 2024 à 16:28, Harry Sarson
<ha...@codethink.co.uk.invalid> a écrit :
> Bazel can be made to fetch sources from a repository cache[2], and bazel
I feel that this is a hack, and we would probably have a better chance
by using a remote cache instead. Please stay tuned for another post to
this list.
> picks
> resources from the repository cache based on both the checksum and the
> "URL that
> bazel thinks the resource lives at" (this is typically the upstream URL). So
> when the revamped bazel_source plugin creates a repository cache for
> bazel it
> needs to know the URL bazel thinks the resource lives at.
> Currently we require the element's author to explicitly add the "URL bazel
> thinks the resource lives at" but if a `Source.get_urls(self) ->
> List[str]` API
> existed then (in 90% of cases where "the URL bazel thinks the resource
> lives at"
> is the upstream URL) the bazel_plugin could determine the URL atomically
> (we may
> not want to implicitly set it, instead adding it during `bst track`).
I am not sure I follow. A `Source.get_urls()` would be implemented by
your `bazel_source` (source) plugin? And then you'd expect a bazel
(element) plugin to call this method and do something with it?
Or are you expecting something else to implement `Source.get_urls()`?
(an earlier source that you place before your bazel_source?). In that
case, please be aware that you cannot get the plugin object of
previous sources: you only get the content that they stage.
Cheers,
Abderrahim
Re: Extending Sources public APIs
Posted by Harry Sarson <ha...@codethink.co.uk.INVALID>.
Hi,
I have been working on a revamped[1] bazel_source plugin to get
buildstream to
download the resources bazel needs to build (bazel normally fetches
resources
from the internet but cannot do that in a sandbox).
Bazel can be made to fetch sources from a repository cache[2], and bazel
picks
resources from the repository cache based on both the checksum and the
"URL that
bazel thinks the resource lives at" (this is typically the upstream URL). So
when the revamped bazel_source plugin creates a repository cache for
bazel it
needs to know the URL bazel thinks the resource lives at.
Currently we require the element's author to explicitly add the "URL bazel
thinks the resource lives at" but if a `Source.get_urls(self) ->
List[str]` API
existed then (in 90% of cases where "the URL bazel thinks the resource
lives at"
is the upstream URL) the bazel_plugin could determine the URL atomically
(we may
not want to implicitly set it, instead adding it during `bst track`).
TLDR; the `Source.get_urls(self) -> List[str]` API sounds very useful to me!
Harry
[1]: the existing bazel_source plugin only supports `http_archive`, is
hard to
configure, and doesn't use buildstream's aliased urls instead
using "URL
that bazel thinks the resource lives at" directly.
[2]:
https://sluongng.hashnode.dev/bazel-caching-explained-pt-3-repository-cache
On 21/11/2024 10:42, Martín Abente Lahaye wrote:
> Hello everyone,
>
> Currently, some of our community plugins like collect_manifest [1] and
> tools like bst-to-lorry [2], rely on a combination of assumptions
> based on the reported “kind” of the Source and private Python APIs.
>
> This can be problematic as both collect_manifest and bst-to-lorry
> query sources for their "kind" and assume that certain attributes and
> methods will be present (e.g., source.url). In fact, this has been
> discussed before at least once [3].
>
> Although this seems to work, it’s unreliable because even if the
> “kind” string matches, there’s no guarantee that it is the expected
> Source, as it can be a different Source with the same “kind” string.
> Plus, even if it really is the expected Source, a future refactor
> could break these assumptions as these aren’t public APIs.
>
> Ultimately, what both of these are trying to determine is:
>
> * The full upstream URL to the source, e.g.,
> https://sourceware.org/git/glibc.git, to include it in the output
> manifest or the lorry configuration file.
> * The version of each source, e.g., 2.40, to include it in the output
> manifest.
> * Other information such as what the source is configured to track, to
> include it in the lorry configuration file.
>
> Therefore, instead of letting plugins and tools do all that unreliable
> guessing, we could provide what these ultimately need by adding new
> abstract methods that each Source can implement. For example,
> something like:
>
> * Source.get_urls(self) -> List[str]: Which would provide a list of
> full upstream URL without any guessing or relying on accessing private
> attributes, for the caller.
> * Source.get_versions(self) -> List[str]: Similarly, for the versions,
> but the tricky piece with this would be the need for a regexp for each
> source, e.g., in case the version needs to be extracted from the
> Source URL.
> * Source.get_trackings(self) -> List[Optional[str]]: Similarly, for
> the tracking strings. Of course this would only make sense for sources
> that can actually be tracked.
>
> Or perhaps, something that better groups these tuples, e.g.,
> Source.get_actual_sources(self) -> List[Tuple[str, str,
> Optional[str]]], providing URL, version and tracking strings tuples or
> equivalent object.
>
> A key question here is whether something like the above would still be
> too rigid or over-specified for that plugin and tool, and perhaps we
> should be thinking of a more free-form API to query for these.
>
> An idea that was mentioned when discussing this topic with Abderrahim
> was to introduce to sources something similar to the elements public
> data, e.g., this way we could add that version matching regexp.
>
> What do you all think?
>
> Regards,
> Martín.
>
> Refs:
> [1]
> https://gitlab.com/BuildStream/buildstream-plugins-community/-/blob/22023ad60e91ff3f635c556ed4c32ce4dfd7c2b5/src/buildstream_plugins_community/elements/collect_manifest.py
> [2]
> https://gitlab.com/CodethinkLabs/lorry/bst-to-lorry/-/blob/d6d3782071502c56611ceffa574d2f81e2a1eedd/bst_to_lorry.py
> [3]
> https://gitlab.com/BuildStream/buildstream-plugins-community/-/issues/2
>
Re: Extending Sources public APIs
Posted by Tristan van Berkom <tr...@codethink.co.uk.INVALID>.
Hi all,
I've been giving this some thought last week, and threw up this pull
request mostly as a draft / RFC.
https://github.com/apache/buildstream/pull/1997
Copied from the PR:
Some thoughts about this approach:
* This prioritizes machine readability and standardization of source
provenance and version information
Sources have a lot of freedom in how they implement things, and so we
may very well need to expand on the types and constants added here,
such as `SourceInfoMedium`, `SourceVersionType`, etc.
The idea here is to have greater certainty about how sources are
obtained, even if this cannot be covered by all currently existing
source implementations (e.g. I didn't initially add a `bzr` medium
for which we have a plugin, or a `cvs` medium for which we do not yet
have a plugin).
Aspirationally, forcing this data to be precise can allow adjacent
tooling to do useful things.
* This drops the freeform "public data" mentioned in the proposal
discussion
My rationale for this choice in this branch, is that ultimately we
want a data with a constant shape, and if for examlpe, we want the
user to be able to override or assist a source with determining the
reported *"version"*, then the `Source` implementation already has
everything it needs to do so:
* it can add additional configuration keys for the user to
configure
* it has the power to implement `collect_source_info()` however it
wants.
* This does not yet attempt to cover the concept of *tracking*
information
I would like to consider this, but we should think carefully about
how this can be useful. For instance, some git plugins have different
interpretations of what their "tracking" strings mean, sometimes
following a branch head, sometimes looking for the latest tag in
history which matches a given regular expression.
If we export this tracking information, it should probably be useful
for external tooling to figure out how to do the tracking and come to
the same conclusion, otherwise it is unclear what this is useful for.
* This does not cover the CVE information
While the SourceInfo objects representing a source's provenance is a
list, I believe that the CVE information continues to be a
per-element concept.
For example, when we have applied security patches to a module, those
security patches are, themselves, sources, with provenance of being
revisioned in the local project
Thoughts and input appreciated,
Cheers,
-Tristan
On Wed, 2024-12-11 at 19:56 +0900, Tristan van Berkom wrote:
> Hi Martín,
>
> On Thu, 2024-11-21 at 07:42 -0300, Martín Abente Lahaye wrote:
> > Hello everyone,
> >
> > Currently, some of our community plugins like collect_manifest [1]
> > and
> > tools like bst-to-lorry [2], rely on a combination of assumptions
> > based
> > on the reported “kind” of the Source and private Python APIs.
> >
> > This can be problematic as both collect_manifest and bst-to-lorry
> > query
> > sources for their "kind" and assume that certain attributes and
> > methods
> > will be present (e.g., source.url). In fact, this has been
> > discussed
> > before at least once [3].
> >
> > Although this seems to work, it’s unreliable because even if the
> > “kind”
> > string matches, there’s no guarantee that it is the expected
> > Source, as
> > it can be a different Source with the same “kind” string. Plus,
> > even if
> > it really is the expected Source, a future refactor could break
> > these
> > assumptions as these aren’t public APIs.
>
> Yes, well known issue, thanks for bringing this back to light :)
>
> [...]
> > Therefore, instead of letting plugins and tools do all that
> > unreliable
> > guessing, we could provide what these ultimately need by adding
> > new
> > abstract methods that each Source can implement. For example,
> > something
> > like:
> >
> > * Source.get_urls(self) -> List[str]: Which would provide a list of
> > full
> > upstream URL without any guessing or relying on accessing private
> > attributes, for the caller.
> > * Source.get_versions(self) -> List[str]: Similarly, for the
> > versions,
> > but the tricky piece with this would be the need for a regexp for
> > each
> > source, e.g., in case the version needs to be extracted from the
> > Source
> > URL.
> > * Source.get_trackings(self) -> List[Optional[str]]: Similarly, for
> > the
> > tracking strings. Of course this would only make sense for sources
> > that
> > can actually be tracked.
>
> I'm very much in favor of crafting Source APIs for Sources to report
> common information about Sources in a standard API path.
>
> This has the advantage of being easy enough to implement on Source
> implementations once, and therefore be leveraged on a multitude of
> projects with little or no effort (asides from perhaps having those
> projects *use* the new plugin versions which support these new APIs).
>
> Into some specifics:
>
> * Given that a Source may have multiple URIs, refs, and tracking
> informations, I think probably a more natural API would be to have
> something like a SourceInfo object defined, and ask Source
> implementations to return a list of them (e.g. list_source_info())
>
> This is mostly just for a pretty API, it's easier this way to know
> what information belongs together.
>
> In 99% of cases, this will return a single entry list.
>
> * Source.get_versions() is ambiguous to me.
>
> From the BuildStream perspective, what is a "version" of a Source
> ?
>
> Is it necessary to know the "version" of a Source without having
> the
> Source data cached locally to compute it ?
>
> It seems to me that we probably want to fetch the source first, so
> that the Source implementation has the liberty to interrogate the
> data in order to guess what a "version" is, perhaps by invoking
> things like `git describe`
>
> I'm not convinced that deriving this information from the URL
> alone,
> or even the URL and the ref, is sufficient for the Source to make
> a qualified "guess" at the version.
>
> In any case, this will likely be a "guess" no matter how the
> plugin computes this "version".
>
>
> > Or perhaps, something that better groups these tuples, e.g.,
> > Source.get_actual_sources(self) -> List[Tuple[str, str,
> > Optional[str]]],
> > providing URL, version and tracking strings tuples or equivalent
> > object.
>
> Ah yes, as I mentioned above, however I would prefer BuildStream
> qualified objects, which are extensible in the future, with some
> versioning strategy, rather than rigidly defining a tuple.
>
> > A key question here is whether something like the above would still
> > be
> > too rigid or over-specified for that plugin and tool, and perhaps
> > we
> > should be thinking of a more free-form API to query for these.
> >
> > An idea that was mentioned when discussing this topic with
> > Abderrahim
> > was to introduce to sources something similar to the elements
> > public
> > data, e.g., this way we could add that version matching regexp.
>
> I think we *also* want public data for this.
>
> I.e. the Source cannot make a fully qualified "guess" at the
> "version",
> and there may be other attributes we want to associate with a
> "source".
>
> As such, the project author should have the authority to override the
> Source implementation's "guess" by explicitly mentioning it in the
> bst
> file.
>
> Further, I would consider this proposal incomplete without adding
> CLI
> support to interrogate this information with.
>
> For the collect_manifest problem, I hope that we can address this
> without needing to use a plugin at all, we should be able to get away
> with only:
>
> * bst source fetch --deps all <ELEMENT>
> * bst source show --deps all --format <FORMAT> <ELEMENT>
>
> Cheers,
> -Tristan
>
>
>
>
>
Re: Extending Sources public APIs
Posted by Tristan van Berkom <tr...@codethink.co.uk.INVALID>.
Hi Martín,
On Thu, 2024-11-21 at 07:42 -0300, Martín Abente Lahaye wrote:
> Hello everyone,
>
> Currently, some of our community plugins like collect_manifest [1] and
> tools like bst-to-lorry [2], rely on a combination of assumptions based
> on the reported “kind” of the Source and private Python APIs.
>
> This can be problematic as both collect_manifest and bst-to-lorry query
> sources for their "kind" and assume that certain attributes and methods
> will be present (e.g., source.url). In fact, this has been discussed
> before at least once [3].
>
> Although this seems to work, it’s unreliable because even if the “kind”
> string matches, there’s no guarantee that it is the expected Source, as
> it can be a different Source with the same “kind” string. Plus, even if
> it really is the expected Source, a future refactor could break these
> assumptions as these aren’t public APIs.
Yes, well known issue, thanks for bringing this back to light :)
[...]
>
> Therefore, instead of letting plugins and tools do all that unreliable
> guessing, we could provide what these ultimately need by adding new
> abstract methods that each Source can implement. For example, something
> like:
>
> * Source.get_urls(self) -> List[str]: Which would provide a list of full
> upstream URL without any guessing or relying on accessing private
> attributes, for the caller.
> * Source.get_versions(self) -> List[str]: Similarly, for the versions,
> but the tricky piece with this would be the need for a regexp for each
> source, e.g., in case the version needs to be extracted from the Source
> URL.
> * Source.get_trackings(self) -> List[Optional[str]]: Similarly, for the
> tracking strings. Of course this would only make sense for sources that
> can actually be tracked.
I'm very much in favor of crafting Source APIs for Sources to report
common information about Sources in a standard API path.
This has the advantage of being easy enough to implement on Source
implementations once, and therefore be leveraged on a multitude of
projects with little or no effort (asides from perhaps having those
projects *use* the new plugin versions which support these new APIs).
Into some specifics:
* Given that a Source may have multiple URIs, refs, and tracking
informations, I think probably a more natural API would be to have
something like a SourceInfo object defined, and ask Source
implementations to return a list of them (e.g. list_source_info())
This is mostly just for a pretty API, it's easier this way to know
what information belongs together.
In 99% of cases, this will return a single entry list.
* Source.get_versions() is ambiguous to me.
From the BuildStream perspective, what is a "version" of a Source ?
Is it necessary to know the "version" of a Source without having the
Source data cached locally to compute it ?
It seems to me that we probably want to fetch the source first, so
that the Source implementation has the liberty to interrogate the
data in order to guess what a "version" is, perhaps by invoking
things like `git describe`
I'm not convinced that deriving this information from the URL alone,
or even the URL and the ref, is sufficient for the Source to make
a qualified "guess" at the version.
In any case, this will likely be a "guess" no matter how the
plugin computes this "version".
>
> Or perhaps, something that better groups these tuples, e.g.,
> Source.get_actual_sources(self) -> List[Tuple[str, str, Optional[str]]],
> providing URL, version and tracking strings tuples or equivalent object.
Ah yes, as I mentioned above, however I would prefer BuildStream
qualified objects, which are extensible in the future, with some
versioning strategy, rather than rigidly defining a tuple.
>
> A key question here is whether something like the above would still be
> too rigid or over-specified for that plugin and tool, and perhaps we
> should be thinking of a more free-form API to query for these.
>
> An idea that was mentioned when discussing this topic with Abderrahim
> was to introduce to sources something similar to the elements public
> data, e.g., this way we could add that version matching regexp.
I think we *also* want public data for this.
I.e. the Source cannot make a fully qualified "guess" at the "version",
and there may be other attributes we want to associate with a "source".
As such, the project author should have the authority to override the
Source implementation's "guess" by explicitly mentioning it in the bst
file.
Further, I would consider this proposal incomplete without adding CLI
support to interrogate this information with.
For the collect_manifest problem, I hope that we can address this
without needing to use a plugin at all, we should be able to get away
with only:
* bst source fetch --deps all <ELEMENT>
* bst source show --deps all --format <FORMAT> <ELEMENT>
Cheers,
-Tristan