You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildstream.apache.org by Sander Striker <st...@apache.org> on 2020/07/30 21:18:14 UTC
Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Hi,
On Thu, Jul 30, 2020 at 11:45 AM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
[...]
> Regarding:
>
> "ways to identify elements that do or do not match the same kind as
> the plugin"
>
> I think this pertains to the Bazel plugin issue you discuss below,
> otherwise I'm not sure what you mean by this :)
>
That was what I meant :).
[...]
> > I've sadly stalled on progress on a bazel plugin[5] that I started for
> > research/discovery reasons as outlined in Chandan's summary message[0].
> > However, even in its current very WIP form, it raises some questions on
> how
> > to do certain things. Like filter all dependencies of the same kind, to
> > stage them differently. I imagine that some of this work would end up
> > being split between stage() and configure_dependencies() following your
> > proposal?
>
> Filtering elements based on the same "kind" is a misguided approach, as
> the element "kind" is not indicative of the nature of the actual plugin
> being used.
>
I am looking for something like an identity function, and not arbitrary
kinds. That is, I want to know if elements are of the same kind or not.
This is the only thing I could come up with so far for that purpose:
class BazelElement(BuildElement):
...
def _is_bazel_kind(self, element: "Element"):
# This element (self) is of the bazel kind, use that to test
return self.get_kind() == element.get_kind()
> This has come up several times since, and has been pointed out on the
> infamous issue regarding plugins accessing unstable plugin API:
>
> https://gitlab.com/BuildStream/bst-plugins-experimental/-/issues/2
>
> I.e. you cannot assume which plugin implements a given "kind", only the
> declaring project knows what it used for a given "kind".
>
> Interestingly, Ben and I have discussed this fairly recently on IRC and
> we do agree that it would be useful for plugins to have some kind of
> identity encoded into them which could be used for this kind of
> conditional treatment of dependencies. I imagine a kind of domain name
> format might be suitable for unique plugin identities, e.g.:
>
> "org.buildstream.git" <-- indicates the "git" plugin maintained by
> BuildStream
>
> That said, I imagine that for the Bazel use case you describe, an
> identity filtering would be more user friendly, I'm not certain of all
> the details but I think you really want to filter dependencies based on
> a specific plugin identity rather than requiring the user to explicitly
> decide which dependencies get treated a certain way.
>
That's exactly right.
> If this is the case, and imagining that we already have something like
> "Plugin.get_identity()", then we could achieve this kind of filtering
> with a simple list comprehension:
>
> class Bazel(Element):
>
> ...
>
> # To be defined: probably a plugin identity would be encoded
> # directly into the plugin class data.
> #
> BST_PLUGIN_IDENTITY = "com.google.bazel"
>
> ...
>
> def configure(self, node):
>
> ...
>
> # Collect the bazel dependencies
> #
> self.bazel_dependencies = [
> element
> for element in self.dependencies(Scope.BUILD)
> if element.get_identity() == "com.google.bazel"
> ]
>
Couldn't we solve this in a more generic way without having to specify a
BST_PLUGIN_IDENTITY which now would need to guarantee uniqueness etc?
Something like the following on Element:
def is_same_kind(self, element):
return self.__class__ is element.__class__
That should do the trick?
Cheers,
Sander
Re: Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Posted by Chandan Singh <ch...@chandansingh.net>.
Heya!
Just a quick inline reply to clarify one point.
> > > I'm not necessarily against having some sort of identity function.
> > > Although, for purposes like this, I wonder if an interface would be
> > > better suited than identity function. Just to throw this idea out
> > > there.
> > >
> > > Instead of doing conditional checks in the would-be Bazel plugin,
> > > perhaps it could define an interface, saying that all bazel elements
> > > need to write something in their public data. So, rather than checking
> > > the element plugin's identity, Bazel plugin would read that public
> > > data and make necessary assumptions.
> >
>
> When you say "that all bazel elements need to write something in their
> public data" do you mean that the plugin would write that public data in
> its artifact, and read it again when consuming such and artifact as a
> dependency?
Exactly. The bazel plugin will first define some sort of interface on
what it will expect from the public data AND be responsible for
encoding the same for elements that are of the bazel kind.
After that, if some other bazel-like plugin wants to pretend as a
bazel plugin, it can also write similar public data to elements of its
kind. And, the bazel plugin will happily consume artifacts of
bazel-like plugin, same as it would consume artifacts of a bazel
plugin.
Again, this is a bit of a contrived example and it'll be good to
understand other use cases as well. But, I think we can get away with
reading/writing public data using plugin-defined interfaces rather
than having the need for an identity function (although that may be
needed for other reasons).
- Chandan
Re: Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Posted by Sander Striker <st...@apache.org>.
Hi,
Thanks for the engaging replies :).
On Fri, Jul 31, 2020 at 2:27 PM Tristan Van Berkom <
tristan.vanberkom@codethink.co.uk> wrote:
> Hi Chandan,
>
> On Fri, 2020-07-31 at 13:02 +0100, Chandan Singh wrote:
> > Hi Sander,
[...]
> > I'm not necessarily against having some sort of identity function.
> > Although, for purposes like this, I wonder if an interface would be
> > better suited than identity function. Just to throw this idea out
> > there.
> >
> > Instead of doing conditional checks in the would-be Bazel plugin,
> > perhaps it could define an interface, saying that all bazel elements
> > need to write something in their public data. So, rather than checking
> > the element plugin's identity, Bazel plugin would read that public
> > data and make necessary assumptions.
>
When you say "that all bazel elements need to write something in their
public data" do you mean that the plugin would write that public data in
its artifact, and read it again when consuming such and artifact as a
dependency?
> If it's not a Bazel plugin, it would lack that public data section, so
> > Bazel plugin could treat that as a normal element.
> >
> > This is a small difference implementation-wise, but I think this
> > approach can lend itself towards more generic solutions. This is a
> > hypothetical example, but bear with me for a moment. Imagine there are
> > two similar plugins - bazel and experimental-bazel. They are largely
> > the same but maybe one of them implements a special performance
> > optimization. Using identities would mean that these two plugins won't
> > be able to take advantage of the optimization based on being of a
> > bazel kind, or would have to have the same identity (which seems a bit
> > wrong).
>
I think that's fair.
> Alternatively, with using public data, both plugins could write
> > necessary markers to some predefined public data domain.
>
> > Let me know what you think.
>
That sounds like an interesting approach. I'll dig a bit to see how to
make that work. It sounds promising.
> Pretty much the same thing crossed my mind when originally replying, I
> think this can be done today without much hassle.
>
> That said, baring in mind that we might want plugin identities for
> other reasons, it might be a better approach to use plugin
> identities... on the other hand, we might find that any requirement
> that would be satisfied by identities would also be satisfied by public
> data markers.
>
> Lets first collect all the pertinent use cases and see what is the
> better solution :)
>
+1. Curious about the other use cases.
> Cheers,
> -Tristan
>
Cheers,
Sander
Re: Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi Chandan,
On Fri, 2020-07-31 at 13:02 +0100, Chandan Singh wrote:
> Hi Sander,
[...]
> > Couldn't we solve this in a more generic way without having to specify a
> > BST_PLUGIN_IDENTITY which now would need to guarantee uniqueness etc?
> >
> > Something like the following on Element:
> >
> > def is_same_kind(self, element):
> > return self.__class__ is element.__class__
> >
> > That should do the trick?
>
> I'm not necessarily against having some sort of identity function.
> Although, for purposes like this, I wonder if an interface would be
> better suited than identity function. Just to throw this idea out
> there.
>
> Instead of doing conditional checks in the would-be Bazel plugin,
> perhaps it could define an interface, saying that all bazel elements
> need to write something in their public data. So, rather than checking
> the element plugin's identity, Bazel plugin would read that public
> data and make necessary assumptions.
>
> If it's not a Bazel plugin, it would lack that public data section, so
> Bazel plugin could treat that as a normal element.
>
> This is a small difference implementation-wise, but I think this
> approach can lend itself towards more generic solutions. This is a
> hypothetical example, but bear with me for a moment. Imagine there are
> two similar plugins - bazel and experimental-bazel. They are largely
> the same but maybe one of them implements a special performance
> optimization. Using identities would mean that these two plugins won't
> be able to take advantage of the optimization based on being of a
> bazel kind, or would have to have the same identity (which seems a bit
> wrong).
>
> Alternatively, with using public data, both plugins could write
> necessary markers to some predefined public data domain.
>
> Let me know what you think.
Pretty much the same thing crossed my mind when originally replying, I
think this can be done today without much hassle.
That said, baring in mind that we might want plugin identities for
other reasons, it might be a better approach to use plugin
identities... on the other hand, we might find that any requirement
that would be satisfied by identities would also be satisfied by public
data markers.
Lets first collect all the pertinent use cases and see what is the
better solution :)
Cheers,
-Tristan
Re: Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Posted by Chandan Singh <ch...@chandansingh.net>.
Hi Sander,
> I am looking for something like an identity function, and not arbitrary
> kinds. That is, I want to know if elements are of the same kind or not.
> This is the only thing I could come up with so far for that purpose:
>
> class BazelElement(BuildElement):
> ...
> def _is_bazel_kind(self, element: "Element"):
> # This element (self) is of the bazel kind, use that to test
> return self.get_kind() == element.get_kind()
>
>
> > This has come up several times since, and has been pointed out on the
> > infamous issue regarding plugins accessing unstable plugin API:
> >
> > https://gitlab.com/BuildStream/bst-plugins-experimental/-/issues/2
> >
> > I.e. you cannot assume which plugin implements a given "kind", only the
> > declaring project knows what it used for a given "kind".
<snip>
> > If this is the case, and imagining that we already have something like
> > "Plugin.get_identity()", then we could achieve this kind of filtering
> > with a simple list comprehension:
> >
> > class Bazel(Element):
> >
> > ...
> >
> > # To be defined: probably a plugin identity would be encoded
> > # directly into the plugin class data.
> > #
> > BST_PLUGIN_IDENTITY = "com.google.bazel"
> >
> > ...
> >
> > def configure(self, node):
> >
> > ...
> >
> > # Collect the bazel dependencies
> > #
> > self.bazel_dependencies = [
> > element
> > for element in self.dependencies(Scope.BUILD)
> > if element.get_identity() == "com.google.bazel"
> > ]
> >
>
> Couldn't we solve this in a more generic way without having to specify a
> BST_PLUGIN_IDENTITY which now would need to guarantee uniqueness etc?
>
> Something like the following on Element:
>
> def is_same_kind(self, element):
> return self.__class__ is element.__class__
>
> That should do the trick?
I'm not necessarily against having some sort of identity function.
Although, for purposes like this, I wonder if an interface would be
better suited than identity function. Just to throw this idea out
there.
Instead of doing conditional checks in the would-be Bazel plugin,
perhaps it could define an interface, saying that all bazel elements
need to write something in their public data. So, rather than checking
the element plugin's identity, Bazel plugin would read that public
data and make necessary assumptions.
If it's not a Bazel plugin, it would lack that public data section, so
Bazel plugin could treat that as a normal element.
This is a small difference implementation-wise, but I think this
approach can lend itself towards more generic solutions. This is a
hypothetical example, but bear with me for a moment. Imagine there are
two similar plugins - bazel and experimental-bazel. They are largely
the same but maybe one of them implements a special performance
optimization. Using identities would mean that these two plugins won't
be able to take advantage of the optimization based on being of a
bazel kind, or would have to have the same identity (which seems a bit
wrong).
Alternatively, with using public data, both plugins could write
necessary markers to some predefined public data domain.
Let me know what you think.
Cheers,
Chandan
Re: Element kind identity matching, WAS: Re: Proposal: Extending
dependency attributes
Posted by Tristan Van Berkom <tr...@codethink.co.uk>.
Hi again,
Nice to kickstart this thread too which also needs to happen...
On Thu, 2020-07-30 at 23:18 +0200, Sander Striker wrote:
> Hi,
[...]
> Couldn't we solve this in a more generic way without having to specify a
> BST_PLUGIN_IDENTITY which now would need to guarantee uniqueness etc?
Possibly for your specifically stated use case.
I think that Ben had other use cases in mind which would benefit from
being able to identify specific plugins.
I think we need to really summarize what things we want to accomplish
with identifying specific plugins in an element's dependencies, if we
want to go as far as calling APIs provided by those elements which are
outside of the core defined Element/Source APIs in BuildStream, this
could scope creep quite quickly (i.e. we'd need ways to assert versions
of other adjacently loaded plugins on top of their identities, etc, I'm
not sure we want to go down that road).
I think it's best if Ben were to ring in on this and remind me what he
wanted with identities...
> Something like the following on Element:
>
> def is_same_kind(self, element):
> return self.__class__ is element.__class__
>
> That should do the trick?
>
This would only work for identifying element plugin matches within the
_same project_. Even if a subproject had separately loaded exactly the
same plugin from exactly the same origin at exactly the same version,
it would be loaded separately into a different namespace for that
subproject.
Cheers,
-Tristan