Re: Missing copyright notices due to LICENSE change

[Robert Burke <lo...@apache.org>]

I have a PR to make the proposed change https://github.com/apache/beam/pull/15201
Apache Legal says the approach seems fine as well: https://issues.apache.org/jira/browse/LEGAL-582

So this is likely going to be merged in today, and possibly cherry picked into the freshly cut release to get the Go SDK docs visible sooner. (This lets the BPG refer to those links reliably as well).

Thanks all!

On 2021/06/02 01:59:04, Robert Burke <ro...@frantil.com> wrote: 
> A link should be fine. The scanner is a similarity detector so small diffs
> shouldn't cause issues.
> 
> Though 100% agree that removing the requirement for that license at all
> would be preferable all around.
> 
> On Tue, Jun 1, 2021, 6:53 PM Ahmet Altay <al...@google.com> wrote:
> 
> >
> >
> > On Tue, May 25, 2021 at 9:22 AM Robert Burke <ro...@frantil.com> wrote:
> >
> >> The owners at pkg go.dev say they can't properly recognize the python
> >> license (see https://github.com/golang/go/issues/45095) due to the
> >> license being somewhat domineering (a go project could *only* have that
> >> license if it had that license, apparently).
> >>
> >> We also can't do a directory specific solution for the Go code.
> >> Pkg.go.dev is an automated crawler and not human curated so they need to
> >> be more conservative (all general LICENSE files to the root of the repo are
> >> included)
> >>
> >> One suggestion is that instead of appending the license to a single file,
> >> we could separate that license into it's own LICENSE.python file which
> >> wouldn't be picked up by the tooling.
> >> I'll take a look at our various license handling /publishing code too and
> >> make appropriate changes to make sure it's not lost in those cases.
> >>
> >>
> >> Another alternative that might work (I've asked in the issue) is
> >> maintaining the license in a LICENSE file, but moving that content in
> >> particular to the sdks/python directory. But this is less preferable since
> >> we have python code in more places than that.
> >>
> >> My vote is for LICENSE.python, and if there are no objections by the next
> >> release cut I'll put forth the PR. I'd rather not make the change so close
> >> to it, and we can cherry pick it otherwise.
> >>
> >
> > If we break the license, we will still need to package all licenses for
> > python packaging, right? As a matter of fact,
> > https://issues.apache.org/jira/browse/LEGAL-417 was about this addition
> > to Beam the response we got was : "Add the Python license into either the
> > project LICENSE, or in a separate file in the tar.gz and a link from the
> > project LICENSE.". So, I think we need to both package and also link from
> > the project LICENSE. Would the linking part work with go tooling?
> >
> > (I think a better option would be re-writing that one specific function
> > and removing the additional licensing needs. For reference this happened
> > at: https://github.com/apache/beam/pull/6423)
> >
> >
> >> Robert Burke
> >> Beam Go Busybody
> >>
> >> On Wed, Mar 24, 2021, 9:25 AM Robert Burke <ro...@frantil.com> wrote:
> >>
> >>> I'm less concerned about the Go Doc at this point.
> >>>
> >>> 0. The Go SDK is still experimental (I know I know, almost there),
> >>> making it less severe of an issue.
> >>> 1 The interpretation pkg.go.dev does is live with the release.
> >>> 1.a That is, if they start accepting the python 2.0 license blob that's
> >>> fixed for all affected versions.
> >>> 1.b Further, if we do cherrypick licence textual fixes into the older
> >>> versions for the licence matching, then the site will pick those up again.
> >>> 2. Users can run the godoc tool
> >>> <https://pkg.go.dev/golang.org/x/tools/cmd/godoc> themselves to get
> >>> package documentation if they want the docs of the version they're using,
> >>> or simply have the programming language server (for go, GoPLS) lift that
> >>> into their IDEs as needed.
> >>>
> >>>
> >>> On Mon, 22 Mar 2021 at 16:42, Ahmet Altay <al...@google.com> wrote:
> >>>
> >>>> For visibility: Change is cherry picked to 2.29.0 release branch (
> >>>> https://github.com/apache/beam/pull/14304).
> >>>>
> >>>> On Mon, Mar 22, 2021 at 12:37 PM Kenneth Knowles <ke...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> So if I understand correctly, the options for a correct license in the
> >>>>> released artifacts are:
> >>>>>
> >>>>>  - revert the change
> >>>>>  - build some automation for bundled jars
> >>>>>  - do something manual?
> >>>>>
> >>>>> Kenn
> >>>>>
> >>>>> On Mon, Mar 22, 2021 at 10:57 AM Brian Hulette <bh...@google.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Pros/cons for a cherrypick:
> >>>>>> (+) Fixes regression for licenses in released Java artifacts.
> >>>>>> (-) It's possible it will permanently break docs on pkg.go.dev for
> >>>>>> 2.29.0 if https://github.com/golang/go/issues/45095 requires changes
> >>>>>> on our end (e.g. fixing the PSF License text).
> >>>>>>
> >>>>>> My sense is the pro outweighs the con here, but I could be convinced
> >>>>>> otherwise. I guess that makes me +0 for cherrypick.
> >>>>>>
> >>>>>> Brian
> >>>>>>
> >>>>>> On Mon, Mar 22, 2021 at 10:43 AM Ahmet Altay <al...@google.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Mar 22, 2021 at 10:31 AM Kenneth Knowles <ke...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Is there a Jira marked as blocking 2.29.0 for the cherrypick?
> >>>>>>>>
> >>>>>>>
> >>>>>>> I do not think so. I have not filed a jira or started a cherry pick
> >>>>>>> pr.
> >>>>>>>
> >>>>>>> Sorry, I was not sure if we agreed to cherry pick or not. Do you
> >>>>>>> want me to do that?
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Mar 19, 2021 at 6:16 PM Valentyn Tymofieiev <
> >>>>>>>> valentyn@google.com> wrote:
> >>>>>>>>
> >>>>>>>>> I also noticed (with a help of an automated tool) that
> >>>>>>>>> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/resources/NOTICES
> >>>>>>>>> includes additional licenses not included in
> >>>>>>>>> https://github.com/apache/beam/blob/master/LICENSE. Is that WAI
> >>>>>>>>> since Dataflow runner is released as a separate jar artifact, and the
> >>>>>>>>> licenses in question (GPL 2.0, CDDL) pertain to its dependencies, or we
> >>>>>>>>> need to include those licenses as well?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 18, 2021 at 9:51 AM Ahmet Altay <al...@google.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 18, 2021 at 6:39 AM Brian Hulette <
> >>>>>>>>>> bhulette@google.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks Robert! I'm +1 for reverting and engaging pkg.go.dev
> >>>>>>>>>>>
> >>>>>>>>>>> > and probably cherry pick it into the affected release branches.
> >>>>>>>>>>> Even if we do this, the Java artifacts from the affected
> >>>>>>>>>>> releases are missing the additional LICENSE text.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> IMO we can skip the cherry picks perhaps with the exception of
> >>>>>>>>>> the upcoming 2.29 release.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> > I do not know how to interpret this ASF guide. As an example
> >>>>>>>>>>> from another project: airflow also has a LICENSE file, NOTICE file, and a
> >>>>>>>>>>> licenses directory. There are even overlapping mentions.
> >>>>>>>>>>> Agreed. I am a software engineer, not a lawyer, and even the
> >>>>>>>>>>> ASF's guide that presumably targets engineers is not particularly clear to
> >>>>>>>>>>> me. This was just my tenuous understanding after a quick review.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Agreed. We can ask LEGAL for further clarification.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Mar 17, 2021 at 7:49 PM Ahmet Altay <al...@google.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thank you Rebo. I agree with reverting first and then figure
> >>>>>>>>>>>> out the next steps.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here is a PR to revert your change:
> >>>>>>>>>>>> https://github.com/apache/beam/pull/14267
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Mar 17, 2021 at 4:02 PM Robert Burke <
> >>>>>>>>>>>> robert@frantil.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Looking at the history it seems that before the python text
> >>>>>>>>>>>>> was added, pkg.go.dev can parse the license stack just fine.
> >>>>>>>>>>>>> It doesn't recognize the PSF license, and fails closed entirely as a result.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I've filed an issue with pkg.go.dev (
> >>>>>>>>>>>>> https://github.com/golang/go/issues/45095). If the bug is
> >>>>>>>>>>>>> fixed, the affected versions will become visible as well.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In the meantime, we should revert my change which clobbered
> >>>>>>>>>>>>> the other licenses and probably cherry pick it into the affected release
> >>>>>>>>>>>>> branches.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The PSF license is annoying as it's explicitly unique. Nothing
> >>>>>>>>>>>>> but python can use it and call it the PSF license. However it is a
> >>>>>>>>>>>>> redistribution friendly license, which is what matters.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Mar 17, 2021, 3:00 PM Ahmet Altay <al...@google.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you for this email.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Mar 17, 2021 at 2:32 PM Brian Hulette <
> >>>>>>>>>>>>>> bhulette@google.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I just noticed that there was a recent change to our LICENSE
> >>>>>>>>>>>>>>> file to make it exactly match the Apache 2.0 License [1]. This seems to be
> >>>>>>>>>>>>>>> the result of two conflicting LICENSE issues.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Go LICENSE issue: The motivation for [1] was to satisfy
> >>>>>>>>>>>>>>> pkg.go.dev's license policies [2]. Prior to the change our
> >>>>>>>>>>>>>>> documentation didn't show up there [3].
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Java artifact LICENSE issue: The removed text contained
> >>>>>>>>>>>>>>> information relevant to "convenience binary distributions". This text was
> >>>>>>>>>>>>>>> added in [4] as a result of this dev@ thread [5], where we
> >>>>>>>>>>>>>>> noticed that copyright notices were missing in binary artifacts. The
> >>>>>>>>>>>>>>> suggested solution (that we went with) was to just add the information to
> >>>>>>>>>>>>>>> the root (source) LICENSE.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Python distribution is missing both files as well. (
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-1746)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'm not sure that that solution is consistent with this ASF
> >>>>>>>>>>>>>>> guide [6] which states:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> > The LICENSE and NOTICE files must *exactly* represent the
> >>>>>>>>>>>>>>> contents of the distribution they reside in. Only components and resources
> >>>>>>>>>>>>>>> that are actually included in a distribution have any bearing on the
> >>>>>>>>>>>>>>> content of that distribution's NOTICE and LICENSE.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I would argue that *just* Apache 2.0 is the correct text for
> >>>>>>>>>>>>>>> our root (source) LICENSE, and the correct way to deal with binary
> >>>>>>>>>>>>>>> artifacts is to generate per-artifact LICENSE/NOTICE files.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I do not know how to interpret this ASF guide. As an example
> >>>>>>>>>>>>>> from another project: airflow also has a LICENSE file, NOTICE file, and a
> >>>>>>>>>>>>>> licenses directory. There are even overlapping mentions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> So right now the Go issue is fixed, but the Java artifact
> >>>>>>>>>>>>>>> issue has regressed. I can think of two potential solutions to resolve both:
> >>>>>>>>>>>>>>> 1) Restore the "convenience binary distributions"
> >>>>>>>>>>>>>>> information, and see if we can get pkg.go.dev to allow it.
> >>>>>>>>>>>>>>> 2) Add infrastructure to generate LICENSE and NOTICE files
> >>>>>>>>>>>>>>> for Java binary artifacts.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have no idea how we might implement (2) so (1) seems more
> >>>>>>>>>>>>>>> tenable, but less correct since it's adding information not relevant to the
> >>>>>>>>>>>>>>> source release.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Brian
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1] https://github.com/apache/beam/pull/11657
> >>>>>>>>>>>>>>> [2] https://pkg.go.dev/license-policy
> >>>>>>>>>>>>>>> [3]
> >>>>>>>>>>>>>>> https://pkg.go.dev/github.com/apache/beam@v2.21.0+incompatible/sdks/go/pkg/beam
> >>>>>>>>>>>>>>> [4] https://github.com/apache/beam/pull/5461
> >>>>>>>>>>>>>>> [5]
> >>>>>>>>>>>>>>> https://lists.apache.org/thread.html/6ef6630e908147ee83e1f1efd4befbda43efb2a59271c5cb49473103@%3Cdev.beam.apache.org%3E
> >>>>>>>>>>>>>>> [6] https://infra.apache.org/licensing-howto.htmlI'll take
> >>>>>>>>>>>>>>> a look at our various license handling /publishing code too and make
> >>>>>>>>>>>>>>> appropriate changes to make sure it's not lost in those cases.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
>