Re: License blacklisting [Was: license-checking script for BuildStream projects]

[Chandan Singh <ch...@chandansingh.net>]

Hi Tristan,

First off, apologies for the delayed response. But, better late than never :)

A couple of comments inline.

On Fri, Aug 28, 2020 at 10:12 AM Tristan Van Berkom
<tr...@codethink.co.uk> wrote:
>
> Hi Chandan,
>
> On Thu, 2020-08-27 at 22:54 +0100, Chandan Singh wrote:
> > Hi Tristan,
> >
> > I don't have enough background on the FreeDesktop side of things, but
> > a couple of comments inline regarding the BuildStream part of the
> > proposal.
> >
> > <snip>
> >
> > > For a vast portion of open source / free software available in the
> > > wild, this conscious interpretation and decision needs to be made by a
> > > human being.
> > >
> > > I would see this implemented in BuildStream in the following way:
> > >
> > >   * Declare a new "licenses" public data format in the bst public data
> > >     domain[3]
> > >
> > >     This is a place where BuildStream project maintainers can record
> > >     the decided license for the module being built, similar to yocto's
> > >     LICENSE variable[1].
> >
> > Personally, I think BuildStream itself shouldn't care about licenses.
> > It should probably not even know that licenses are a thing. As I see
> > it, BuildStream is a build tool and managing licenses shouldn't be its
> > responsibility. Auditing of sources seems like a separate concern to
> > me.
> >
> > At least the way I use BuildStream, by the time I decide to use a
> > source in a BuildStream element, I have already made a decision to use
> > it and hopefully have done my due diligence and auditing. But, maybe
> > others have different usage patterns. In any case, I feel like
> > BuildStream core shouldn't have an opinion about licenses or how to
> > manage them. But, having a plugin do this job is definitely fair game.
>
> It is especially relevant because BuildStream projects are a shared
> good, this is what junctions essentially solve.
>
> When building something in the BuildStream ecosystem, you are probably
> interacting with upstream junctions which they themselves provide a
> hand full of modules under various licenses.
>
> These upstreams are probably a mix of FOSS upstreams and ISVs.
>
> In the end you may very well trust that these upstreams have done their
> due diligence in attributing the correct licenses to the modules they
> provide you (and these upstreams may even be liable for such depending
> on the contract you have with your upstreams).
>
> At the end of the day, you will probably junction a hand full of
> upstreams, add your own highlevel application on top of it all, and
> ship it.
>
> At this time, it becomes highly valueable that you can simply run an
> assertion element which check for license compatibility of all of the
> modules which ended up in your dependency chain.

Thanks for the summary. I do understand what we are trying to do here,
and I do think BuildStream itself should allow for such an assertion
to happen, This can be done via allowing sufficient API for a plugin
to exist that makes such assertions. I just don't think BuildStream
core itself should have to do anything relating to licenses. Would you
agree with this?

> > >     For compatibility across tooling, and consideration of possible
> > >     further automation (see further below), we should probably assert
> > >     that these license annotations be valid SPDX license
> > >     identifiers[4].
> > >
> > >   * We would add a new Element plugin in BuildStream, and call it
> > >     something like `assertlicense`
> > >
> > >     In this element's `config`, it would allow the user to declare
> > >     a blacklist.
> > >
> > >     This element could output a manifest of licenses in the artifact,
> > >     or produce no output at all, the important part is that this
> > >     element can be added to the pipeline, depend on some elements,
> > >     and halt the build with an error in the case that invalid
> > >     licenses are detected.
> >
> > I don't have anything against such a plugin. But, I'd be much happier
> > if the public data specification was defined by such a plugin rather
> > than BuildStream itself. Maybe a family of plugins decide to follow a
> > shared format, and maybe some other family could decide to do things
> > differently if their needs aren't met. Either way, BuildStream core
> > won't need to change or care.
>
> This raises an interesting point about naming public data which belongs
> to various plugins.
>
> We could theoretically split up the `bst` domain into separate
> categories, keeping only keys which are understood by plugins that are
> wired into the code BuildStream plugins.
>
> I think it makes better sense to have the `bst` domain cover all
> plugins which are maintained under the BuildStream umbrella.
>
> In the end, this is only a question about how to use the `bst`
> namespace.

Indeed, this is mostly a question around the ownership of the `bst`
namespace. My suggestion is that it should be reserved by BuildStream
core itself. That is, if the presence of that plugin data concerns
BuildStream itself in any way.

The current builtin public data fields are: integration-commands,
split-rules and overlap-whitelist. All of these change BuildStream's
behavior in one way or the other. For example, integration-commands is
so much part of the core interface, that it gets its own flag in `bst
artifact checkout --integrate/--no-integrate`.

As such, I think we should keep the `bst` namespace for things
concerning the BuildStream core.

If we agree on this separation of concerns, I'm not sure what would be
the best alternative, i.e. how should we group things together, and
how fine-grained should that be. If we can't come up with a better
scheme, something like `core-plugins` or `bst-plugins` won't be any
worse than `bst` in my opinion.

Cheers,
Chandan

Re: [Freedesktop-sdk] License blacklisting [Was: license-checking script for BuildStream projects]

[Tristan Van Berkom <tr...@codethink.co.uk>]

Hi,

On Tue, 2020-09-01 at 21:08 +0100, Chandan Singh wrote:
> Hi Tristan,
[...]
> Thanks for the summary. I do understand what we are trying to do here,
> and I do think BuildStream itself should allow for such an assertion
> to happen, This can be done via allowing sufficient API for a plugin
> to exist that makes such assertions. I just don't think BuildStream
> core itself should have to do anything relating to licenses. Would you
> agree with this?

Yes.

> Indeed, this is mostly a question around the ownership of the `bst`
> namespace. My suggestion is that it should be reserved by BuildStream
> core itself. That is, if the presence of that plugin data concerns
> BuildStream itself in any way.
> 
> The current builtin public data fields are: integration-commands,
> split-rules and overlap-whitelist. All of these change BuildStream's
> behavior in one way or the other. For example, integration-commands is
> so much part of the core interface, that it gets its own flag in `bst
> artifact checkout --integrate/--no-integrate`.

I should point out that for each of these, these public data were
conceived of for the purpose of consumption by reverse dependency
elements, and for convenience, some of it eventually ended up getting
meshed into core convenience APIs (and even functionality) indeed.

> As such, I think we should keep the `bst` namespace for things
> concerning the BuildStream core.
> 
> If we agree on this separation of concerns, I'm not sure what would be
> the best alternative, i.e. how should we group things together, and
> how fine-grained should that be. If we can't come up with a better
> scheme, something like `core-plugins` or `bst-plugins` won't be any
> worse than `bst` in my opinion.

I think it would make sense to use the `bst-` prefix for any public
data belonging to plugins maintained by BuildStream.

Also, I don't think public data consumed only by plugins needs to be
tied to a specific plugin name, but rather should be named after a
domain / use case (naming the data after what it represents, and not
after what it first happened to be used for, public data annotations
can be reused by multiple plugins).

I think that in a world where we split plugins out of the core (very
near future), all plugins maintained by BuildStream are "upstream
buildstream plugins", and as such, I would not mind sharing the "bst"
namespace directly across those.

While I don't mind so much about using the "bst" name directly, I do
think that BuildStream plugins should either use the "bst" domain
directly, or use the "bst-" prefix for their domain names (i.e.
buildstream plugins should not use un-prefixed public data domain
names).

Cheers,
    -Tristan