License Verification with Weasel

[Chris Lemmons <al...@gmail.com>]

A few months ago, we spent some time cleaning up our licenses for Apache.
In order to find the problem licenses, I wrote a semi-general–purpose tool
for validating the repo for Apache. I put in a PR in February to get it
running automatically in the CI, but the TC repo isn't really the place for
a tool like that.

So I took the tool, renamed it Weasel (it's a cousin to Rat :) ), and
tossed it into a repo all alone, where it can be used as a general purpose
tool. I just created a PR for running Weasel from our docker builds, which
should get them running in our CI. That's great, because it means
significantly increased license scrutiny, all without humans intervening.

Weasel isn't a straight-up replacement for Rat. It's got a lot of features
that Rat doesn't have, specifically around the way we vendor our Go
libraries and licensing for binary files, but Rat integrates directly with
the Apache framework in a nice, visible way, and Rat has more advanced
license checking.

I've got a PR so you can see how it works, but what do y'all think?

PR Link: https://github.com/apache/incubator-trafficcontrol/pull/809
Weasel Link: https://github.com/Comcast/weasel

Re: License Verification with Weasel

[Chris Lemmons <al...@gmail.com>]

FWIW, I went ahead and removed vendoring for weasel from my PR. It now
requires an Internet connection to build and always uses the head. but
it's not in the direct build path, so even if it breaks, you can still
get reliable output.

I do agree that vendoring is a better solution in this case, but it
sounds like I'm a minority voice on that point. There are some
advantages to a dynamic toolchain as well, so it will work fine either
way. This means there's one fewer place to update when there are fixes
to the toolchain.

On Mon, Dec 18, 2017 at 5:38 PM, Robert Butts <ro...@gmail.com> wrote:
> +1 This is immensely valuable, thanks for putting all this work in to fix
> our Trafficcontrol license pains! Sorry about the slow response, looks like
> this is a classic case of Lazy Consensus.
>
>>a particular emphasis on whether vendoring is a suitable solution for
> executables instead of libraries.
>
> Also +1 on vendoring Goose. There's no logical reason vendoring binaries is
> different from source, as long as we include SHA512 hashes for
> verification. And there are big reasons to vendor, primarily security and
> future unavailability. Neither our build, run, nor build checks should fail
> if the internet or some random website is down. A la the Leftpad Fiasco.
>
> I'd support vendoring both for Weasel, and whenever the source is
> Apache-license-compatible.
>
> We've had countless License violations in the last year, that RAT didn't
> catch, that Weasel would. IMO we should merge this ASAP, and get it into
> the Build system, so we can break builds when PRs violate licenses. It will
> save a great deal of time and headache.
>
>
>
> On Fri, Dec 8, 2017 at 8:10 PM, Chris Lemmons <al...@gmail.com> wrote:
>
>> Dan recently suggested I revive this discussion with a particular
>> emphasis on whether vendoring is a suitable solution for executables
>> instead of libraries. He correctly observed that we decided against
>> vendoring goose.
>>
>> I'd like to revisit the topic, since I think it's particularly
>> valuable to be able to build without requiring off-site resources.In
>> goose's case, the executable is required at application run time.
>> weasel, on the other hand, is required at build time (or test time,
>> depending how you see things). On the other hand, github doesn't
>> really go down frequently, and if we always get the latest version, we
>> can be assured of having the latest fixes. Of course, that does mean
>> the possibility of breakage from an external tool, which is what
>> vendoring is usually used to avoid.
>>
>> I'm pro-vendoring in this case. Are there strong opinions either way
>> on this topic?
>>
>> On Mon, Aug 14, 2017 at 5:31 PM, Chris Lemmons <al...@gmail.com> wrote:
>> > A few months ago, we spent some time cleaning up our licenses for
>> Apache. In
>> > order to find the problem licenses, I wrote a semi-general–purpose tool
>> for
>> > validating the repo for Apache. I put in a PR in February to get it
>> running
>> > automatically in the CI, but the TC repo isn't really the place for a
>> tool
>> > like that.
>> >
>> > So I took the tool, renamed it Weasel (it's a cousin to Rat :) ), and
>> tossed
>> > it into a repo all alone, where it can be used as a general purpose
>> tool. I
>> > just created a PR for running Weasel from our docker builds, which should
>> > get them running in our CI. That's great, because it means significantly
>> > increased license scrutiny, all without humans intervening.
>> >
>> > Weasel isn't a straight-up replacement for Rat. It's got a lot of
>> features
>> > that Rat doesn't have, specifically around the way we vendor our Go
>> > libraries and licensing for binary files, but Rat integrates directly
>> with
>> > the Apache framework in a nice, visible way, and Rat has more advanced
>> > license checking.
>> >
>> > I've got a PR so you can see how it works, but what do y'all think?
>> >
>> > PR Link: https://github.com/apache/incubator-trafficcontrol/pull/809
>> > Weasel Link: https://github.com/Comcast/weasel
>>

Re: License Verification with Weasel

[Robert Butts <ro...@gmail.com>]

+1 This is immensely valuable, thanks for putting all this work in to fix
our Trafficcontrol license pains! Sorry about the slow response, looks like
this is a classic case of Lazy Consensus.

>a particular emphasis on whether vendoring is a suitable solution for
executables instead of libraries.

Also +1 on vendoring Goose. There's no logical reason vendoring binaries is
different from source, as long as we include SHA512 hashes for
verification. And there are big reasons to vendor, primarily security and
future unavailability. Neither our build, run, nor build checks should fail
if the internet or some random website is down. A la the Leftpad Fiasco.

I'd support vendoring both for Weasel, and whenever the source is
Apache-license-compatible.

We've had countless License violations in the last year, that RAT didn't
catch, that Weasel would. IMO we should merge this ASAP, and get it into
the Build system, so we can break builds when PRs violate licenses. It will
save a great deal of time and headache.



On Fri, Dec 8, 2017 at 8:10 PM, Chris Lemmons <al...@gmail.com> wrote:

> Dan recently suggested I revive this discussion with a particular
> emphasis on whether vendoring is a suitable solution for executables
> instead of libraries. He correctly observed that we decided against
> vendoring goose.
>
> I'd like to revisit the topic, since I think it's particularly
> valuable to be able to build without requiring off-site resources.In
> goose's case, the executable is required at application run time.
> weasel, on the other hand, is required at build time (or test time,
> depending how you see things). On the other hand, github doesn't
> really go down frequently, and if we always get the latest version, we
> can be assured of having the latest fixes. Of course, that does mean
> the possibility of breakage from an external tool, which is what
> vendoring is usually used to avoid.
>
> I'm pro-vendoring in this case. Are there strong opinions either way
> on this topic?
>
> On Mon, Aug 14, 2017 at 5:31 PM, Chris Lemmons <al...@gmail.com> wrote:
> > A few months ago, we spent some time cleaning up our licenses for
> Apache. In
> > order to find the problem licenses, I wrote a semi-general–purpose tool
> for
> > validating the repo for Apache. I put in a PR in February to get it
> running
> > automatically in the CI, but the TC repo isn't really the place for a
> tool
> > like that.
> >
> > So I took the tool, renamed it Weasel (it's a cousin to Rat :) ), and
> tossed
> > it into a repo all alone, where it can be used as a general purpose
> tool. I
> > just created a PR for running Weasel from our docker builds, which should
> > get them running in our CI. That's great, because it means significantly
> > increased license scrutiny, all without humans intervening.
> >
> > Weasel isn't a straight-up replacement for Rat. It's got a lot of
> features
> > that Rat doesn't have, specifically around the way we vendor our Go
> > libraries and licensing for binary files, but Rat integrates directly
> with
> > the Apache framework in a nice, visible way, and Rat has more advanced
> > license checking.
> >
> > I've got a PR so you can see how it works, but what do y'all think?
> >
> > PR Link: https://github.com/apache/incubator-trafficcontrol/pull/809
> > Weasel Link: https://github.com/Comcast/weasel
>

Re: License Verification with Weasel

[Chris Lemmons <al...@gmail.com>]

Dan recently suggested I revive this discussion with a particular
emphasis on whether vendoring is a suitable solution for executables
instead of libraries. He correctly observed that we decided against
vendoring goose.

I'd like to revisit the topic, since I think it's particularly
valuable to be able to build without requiring off-site resources.In
goose's case, the executable is required at application run time.
weasel, on the other hand, is required at build time (or test time,
depending how you see things). On the other hand, github doesn't
really go down frequently, and if we always get the latest version, we
can be assured of having the latest fixes. Of course, that does mean
the possibility of breakage from an external tool, which is what
vendoring is usually used to avoid.

I'm pro-vendoring in this case. Are there strong opinions either way
on this topic?

On Mon, Aug 14, 2017 at 5:31 PM, Chris Lemmons <al...@gmail.com> wrote:
> A few months ago, we spent some time cleaning up our licenses for Apache. In
> order to find the problem licenses, I wrote a semi-general–purpose tool for
> validating the repo for Apache. I put in a PR in February to get it running
> automatically in the CI, but the TC repo isn't really the place for a tool
> like that.
>
> So I took the tool, renamed it Weasel (it's a cousin to Rat :) ), and tossed
> it into a repo all alone, where it can be used as a general purpose tool. I
> just created a PR for running Weasel from our docker builds, which should
> get them running in our CI. That's great, because it means significantly
> increased license scrutiny, all without humans intervening.
>
> Weasel isn't a straight-up replacement for Rat. It's got a lot of features
> that Rat doesn't have, specifically around the way we vendor our Go
> libraries and licensing for binary files, but Rat integrates directly with
> the Apache framework in a nice, visible way, and Rat has more advanced
> license checking.
>
> I've got a PR so you can see how it works, but what do y'all think?
>
> PR Link: https://github.com/apache/incubator-trafficcontrol/pull/809
> Weasel Link: https://github.com/Comcast/weasel