You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by Dale LaBossiere <dl...@apache.org> on 2017/07/11 19:29:24 UTC
maven and distribution bundles with transitive deps: licensing¬ice
Hi,
Can’t we eliminate releasing binary bundles containing transitive deps and in doing so
eliminate a huge amount of licensing/notice pain for us? But still supply what users need.
As I understand it / ASF policy, the newly generated distribution bundles lack the necessary
license/notice info for transitive deps contained in the bundle.
That’s what the binary-release/LICENSE and licenses/binary-release/* was all about
(included in the gradle generated binary bundles).
Justin indicated we needed to have full copies of license/notice text not just URLs to it
(since the URL may have different text at a later date that doesn’t match that text
applicable for the contained deps).
I see the new distribution stuff also generates edgent-distribution-<ver>.jar.
Its META-INF/DEPENDENCIES has a nice listing of all the transitive deps
license info but it’s (a) just URLs to licenses, (b) lacks any notice info,
and (c) isn’t included in the generated binary release bundle.
Maybe I just don’t understand how this is supposed to work.
Not releasing a binary bundle also eliminates validation/testing of it.
My hope was that viable / better alternative is to provide an easy way for users
themselves to get the Edgent jars and transitive external deps themselves from
maven-central (or any maven repo).
Presumably we'd just need to tell the user something like:
“This command retrieves the Edgent jars and their transitive external dependencies.
The external dependencies come with their own licensing terms that you should review.
A summary of the transitive dependencies and their licenses can be found here
<url to something like the info in the aforementioned META-INF/DEPENDENCIES
and/or binary-release/LICENSING file>.
Continue? <yes|no>”
That’s what get-edgent-jars.sh (https://paste.apache.org/p/GI0n <https://paste.apache.org/p/GI0n>) was working towards.
Is this making sense / compelling / a valid and (sufficiently) user friendly approach?
To clarify for all, this tool (equivalently a binary bundle) is only needed by
Edgent users that don’t use maven/maven-repo enabled app development tools
or that don’t use those tools to generate a standalone “über jar” for their Edgent app
to deploy to their edge device.
— Dale
Re: maven and distribution bundles with transitive deps: licensing¬ice
Posted by "John D. Ament" <jo...@apache.org>.
On Tue, Jul 11, 2017 at 8:10 PM Justin Mclean <ju...@classsoftware.com>
wrote:
> Hi,
>
> > As I understand it / ASF policy, the newly generated distribution
> bundles lack the necessary
> > license/notice info for transitive deps contained in the bundle.
>
> In general dependancies don’t modify LICENSE or NOTICE files only what is
> bundled in an artefact does.
>
Right, when I spoke with Dale and Will at ApacheCon, one of the points that
I made was that they only had a binary distribution which included their
transitive dependencies (individual JARs weren't downloadable). If they
didn't have that binary dist, and instead just used maven coordinates, they
could eliminate the licensing/notices on that artifact by that artifact
simply not existing.
>
> Thanks,
> Justin
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Dale LaBossiere <dm...@gmail.com>.
Yup, the external dependencies are bundled in the convenience binary artifact, hence the L/N mods.
> On Jul 11, 2017, at 8:10 PM, Justin Mclean <ju...@classsoftware.com> wrote:
>
> Hi,
>
>> As I understand it / ASF policy, the newly generated distribution bundles lack the necessary
>> license/notice info for transitive deps contained in the bundle.
>
> In general dependancies don’t modify LICENSE or NOTICE files only what is bundled in an artefact does.
>
> Thanks,
> Justin
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> As I understand it / ASF policy, the newly generated distribution bundles lack the necessary
> license/notice info for transitive deps contained in the bundle.
In general dependancies don’t modify LICENSE or NOTICE files only what is bundled in an artefact does.
Thanks,
Justin
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Dale LaBossiere <dm...@gmail.com>.
Originally, the Edgent/Quarks repo had the actual dependent ext jars as part of it and the bin release bundle copied them from there. Kafka was a pretty large collection of things, many of which were completely unneeded for our use due to the way it was packaged (e.g., its noms pulled in a CLI arg handler, junit, ...). Hence the Edgent repo only included the actual needed jars and their needed deps.
When we went to gradle we retained that logical behavior. The connector’s build.gradle describes the exact jars required, using the “@jar” dependency syntax to exclude transitive deps. There was no expectation/requirement of “provided” on the part of the Edgent consumer.
So that’s the history. If possible/practical, for now I think it best to retain that behavior across the mvn conversion. Make sense?
— Dale
> On Jul 13, 2017, at 2:48 AM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi Dale,
>
> I just tested your script and it worked perfectly :-)
>
> Regarding your observations:
> - You mentioned in another post, that I should exclude any transitive dependency of Kafka … The only valid reason I could see to need this, if using Edgent in an environment in which Kafka is provided by other means. The Maven way to handle such a situation is to mark that dependency “provided” in which case it is used during compilation, but is excluded from being pulled in automatically … like the servlet-api jars which usually are provided by every servlet container.
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
First version of a auto-jar-downloading version:
Am 13.07.17, 08:48 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Dale,
I just tested your script and it worked perfectly :-)
Regarding your observations:
- You mentioned in another post, that I should exclude any transitive dependency of Kafka … The only valid reason I could see to need this, if using Edgent in an environment in which Kafka is provided by other means. The Maven way to handle such a situation is to mark that dependency “provided” in which case it is used during compilation, but is excluded from being pulled in automatically … like the servlet-api jars which usually are provided by every servlet container.
- I’ll investigate why the animal-sniffer-annotations and alike are included … I guess they shouldn’t. The reason for a different Jetty version in Java7, is that the version used in java8 is not compatible with java7, so I had to go back to the newest Jetty version that supports Java7.
Just forked the Maven-Wrapper project and will start working on a version of it that works without a checked in jar. Any other solution (excluding jar from the release) sounds like a hack and it would break the one requirement of Apache releases, that you should be able to download and unpack the release and simply run it to reproduce the same results. I guess the changes to the wrapper needed here are minimal and if we had them all Apache projects could easily use that versions without any problems.
Chris
Am 12.07.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Chris,
I’ve uploaded a new version of get-edgent-jars.sh to https://paste.apache.org/VuU3 <https://paste.apache.org/VuU3>
After you’ve built the j8,j7,android platforms you can do the following to get the jars:
sh get-edgent-jars.sh --platform java8 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform java7 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform android --version 1.2.0-SNAPSHOT
I’m seeing some odd results that I believe are purely a result of what the Edgent poms have in them (I think I have your most recent changes).
Comparing what’s collected for j8 vs j7 vs android…
- with the “provided” that was recently added, the kafka_2.10-0.8.2.2.jar (and deps) are “missing” from j8,j7,android
- j8 vs j7 ext-jars - there are extra jars in j7 like animal-sniffer-annotations-1.14 and others; there are different versions of jetty jars
- j8 vs android ext-jars - similar extra jars in android
— Dale
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok ... that wen’t away too fast …
Here’s the link:
https://github.com/chrisdutz/maven-wrapper/commit/b0487654b032ee59446919f22c01c2235d8c4e93
Hopefully my colleagues will help me make it solid and perfect :-)
Chris
Am 13.07.17, 08:48 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi Dale,
I just tested your script and it worked perfectly :-)
Regarding your observations:
- You mentioned in another post, that I should exclude any transitive dependency of Kafka … The only valid reason I could see to need this, if using Edgent in an environment in which Kafka is provided by other means. The Maven way to handle such a situation is to mark that dependency “provided” in which case it is used during compilation, but is excluded from being pulled in automatically … like the servlet-api jars which usually are provided by every servlet container.
- I’ll investigate why the animal-sniffer-annotations and alike are included … I guess they shouldn’t. The reason for a different Jetty version in Java7, is that the version used in java8 is not compatible with java7, so I had to go back to the newest Jetty version that supports Java7.
Just forked the Maven-Wrapper project and will start working on a version of it that works without a checked in jar. Any other solution (excluding jar from the release) sounds like a hack and it would break the one requirement of Apache releases, that you should be able to download and unpack the release and simply run it to reproduce the same results. I guess the changes to the wrapper needed here are minimal and if we had them all Apache projects could easily use that versions without any problems.
Chris
Am 12.07.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Chris,
I’ve uploaded a new version of get-edgent-jars.sh to https://paste.apache.org/VuU3 <https://paste.apache.org/VuU3>
After you’ve built the j8,j7,android platforms you can do the following to get the jars:
sh get-edgent-jars.sh --platform java8 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform java7 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform android --version 1.2.0-SNAPSHOT
I’m seeing some odd results that I believe are purely a result of what the Edgent poms have in them (I think I have your most recent changes).
Comparing what’s collected for j8 vs j7 vs android…
- with the “provided” that was recently added, the kafka_2.10-0.8.2.2.jar (and deps) are “missing” from j8,j7,android
- j8 vs j7 ext-jars - there are extra jars in j7 like animal-sniffer-annotations-1.14 and others; there are different versions of jetty jars
- j8 vs android ext-jars - similar extra jars in android
— Dale
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Dale,
I just tested your script and it worked perfectly :-)
Regarding your observations:
- You mentioned in another post, that I should exclude any transitive dependency of Kafka … The only valid reason I could see to need this, if using Edgent in an environment in which Kafka is provided by other means. The Maven way to handle such a situation is to mark that dependency “provided” in which case it is used during compilation, but is excluded from being pulled in automatically … like the servlet-api jars which usually are provided by every servlet container.
- I’ll investigate why the animal-sniffer-annotations and alike are included … I guess they shouldn’t. The reason for a different Jetty version in Java7, is that the version used in java8 is not compatible with java7, so I had to go back to the newest Jetty version that supports Java7.
Just forked the Maven-Wrapper project and will start working on a version of it that works without a checked in jar. Any other solution (excluding jar from the release) sounds like a hack and it would break the one requirement of Apache releases, that you should be able to download and unpack the release and simply run it to reproduce the same results. I guess the changes to the wrapper needed here are minimal and if we had them all Apache projects could easily use that versions without any problems.
Chris
Am 12.07.17, 19:37 schrieb "Dale LaBossiere" <dm...@gmail.com>:
Chris,
I’ve uploaded a new version of get-edgent-jars.sh to https://paste.apache.org/VuU3 <https://paste.apache.org/VuU3>
After you’ve built the j8,j7,android platforms you can do the following to get the jars:
sh get-edgent-jars.sh --platform java8 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform java7 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform android --version 1.2.0-SNAPSHOT
I’m seeing some odd results that I believe are purely a result of what the Edgent poms have in them (I think I have your most recent changes).
Comparing what’s collected for j8 vs j7 vs android…
- with the “provided” that was recently added, the kafka_2.10-0.8.2.2.jar (and deps) are “missing” from j8,j7,android
- j8 vs j7 ext-jars - there are extra jars in j7 like animal-sniffer-annotations-1.14 and others; there are different versions of jetty jars
- j8 vs android ext-jars - similar extra jars in android
— Dale
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Dale LaBossiere <dm...@gmail.com>.
Chris,
I’ve uploaded a new version of get-edgent-jars.sh to https://paste.apache.org/VuU3 <https://paste.apache.org/VuU3>
After you’ve built the j8,j7,android platforms you can do the following to get the jars:
sh get-edgent-jars.sh --platform java8 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform java7 --version 1.2.0-SNAPSHOT
sh get-edgent-jars.sh --platform android --version 1.2.0-SNAPSHOT
I’m seeing some odd results that I believe are purely a result of what the Edgent poms have in them (I think I have your most recent changes).
Comparing what’s collected for j8 vs j7 vs android…
- with the “provided” that was recently added, the kafka_2.10-0.8.2.2.jar (and deps) are “missing” from j8,j7,android
- j8 vs j7 ext-jars - there are extra jars in j7 like animal-sniffer-annotations-1.14 and others; there are different versions of jetty jars
- j8 vs android ext-jars - similar extra jars in android
— Dale
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi John,
Is this considered a best-practice at Apache? I remember us having a similar discussion at the Flex project and we settled with not checking in the jar and not using maven wrapper.
Chris
Am 12.07.17, 15:11 schrieb "John D. Ament" <jo...@apache.org>:
On Wed, Jul 12, 2017 at 9:09 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Well the problem is, the wrapper would require us to check in a jar and
> that’s a no-go for an Apache release.
>
you can check in JARs to the repo. Just exclude it from the source release.
>
> There seems to be long-going discussions on the maven-wrapper page about
> such a no-deps update. Right now, I think the most promising solution would
> be to extend the mvnw scripts to check if the jar is present and if it’s
> not, to compile a single java class and run it locally. This would then
> download the jar and place it where the wrapper expects it to be. In that
> case only sources would be checked in and the only dependency needed would
> be to have a JDKs bin directory on the systems path (Which is needed for
> almost everything else anyway).
>
> Unfortunately, I am totally at war with platform (independent)
> shell-scripts. Sort of have the same love for them as for regular
> expressions ;-)
>
> Volunteers to look into this?
>
> Chris
>
>
> Am 12.07.17, 14:34 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> The wrapper sounds like a win (we currently leverage the gradle
> wrapper to ease the pain).
>
> I’m continuing to tweak the script for j7/android.
>
> > On Jul 11, 2017, at 5:07 PM, Christofer Dutz <
> christofer.dutz@c-ware.de> wrote:
> >
> > I'm not demanding the assembly ... It was just the last missing part
> in the migration to Maven. Most ASF projects provide such convenience
> binary archives, but an ASF release is the source only anyway.
> >
> > I admit that your script would do the job.
> >
> > Another thing worth looking into might be the Maven wrapper, which
> would eliminate the need to download Maven, as someone not wanting to use
> Maven would have to install Maven in order to NOT use it afterwards ;-) ...
> Users using Maven wouldn't use the script.
> >
> > Chris
>
>
>
>
Re: maven and distribution bundles with transitive deps: licensing¬ice
Posted by "John D. Ament" <jo...@apache.org>.
On Wed, Jul 12, 2017 at 9:09 AM Christofer Dutz <ch...@c-ware.de>
wrote:
> Well the problem is, the wrapper would require us to check in a jar and
> that’s a no-go for an Apache release.
>
you can check in JARs to the repo. Just exclude it from the source release.
>
> There seems to be long-going discussions on the maven-wrapper page about
> such a no-deps update. Right now, I think the most promising solution would
> be to extend the mvnw scripts to check if the jar is present and if it’s
> not, to compile a single java class and run it locally. This would then
> download the jar and place it where the wrapper expects it to be. In that
> case only sources would be checked in and the only dependency needed would
> be to have a JDKs bin directory on the systems path (Which is needed for
> almost everything else anyway).
>
> Unfortunately, I am totally at war with platform (independent)
> shell-scripts. Sort of have the same love for them as for regular
> expressions ;-)
>
> Volunteers to look into this?
>
> Chris
>
>
> Am 12.07.17, 14:34 schrieb "Dale LaBossiere" <dm...@gmail.com>:
>
> The wrapper sounds like a win (we currently leverage the gradle
> wrapper to ease the pain).
>
> I’m continuing to tweak the script for j7/android.
>
> > On Jul 11, 2017, at 5:07 PM, Christofer Dutz <
> christofer.dutz@c-ware.de> wrote:
> >
> > I'm not demanding the assembly ... It was just the last missing part
> in the migration to Maven. Most ASF projects provide such convenience
> binary archives, but an ASF release is the source only anyway.
> >
> > I admit that your script would do the job.
> >
> > Another thing worth looking into might be the Maven wrapper, which
> would eliminate the need to download Maven, as someone not wanting to use
> Maven would have to install Maven in order to NOT use it afterwards ;-) ...
> Users using Maven wouldn't use the script.
> >
> > Chris
>
>
>
>
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
Well the problem is, the wrapper would require us to check in a jar and that’s a no-go for an Apache release.
There seems to be long-going discussions on the maven-wrapper page about such a no-deps update. Right now, I think the most promising solution would be to extend the mvnw scripts to check if the jar is present and if it’s not, to compile a single java class and run it locally. This would then download the jar and place it where the wrapper expects it to be. In that case only sources would be checked in and the only dependency needed would be to have a JDKs bin directory on the systems path (Which is needed for almost everything else anyway).
Unfortunately, I am totally at war with platform (independent) shell-scripts. Sort of have the same love for them as for regular expressions ;-)
Volunteers to look into this?
Chris
Am 12.07.17, 14:34 schrieb "Dale LaBossiere" <dm...@gmail.com>:
The wrapper sounds like a win (we currently leverage the gradle wrapper to ease the pain).
I’m continuing to tweak the script for j7/android.
> On Jul 11, 2017, at 5:07 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> I'm not demanding the assembly ... It was just the last missing part in the migration to Maven. Most ASF projects provide such convenience binary archives, but an ASF release is the source only anyway.
>
> I admit that your script would do the job.
>
> Another thing worth looking into might be the Maven wrapper, which would eliminate the need to download Maven, as someone not wanting to use Maven would have to install Maven in order to NOT use it afterwards ;-) ... Users using Maven wouldn't use the script.
>
> Chris
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Dale LaBossiere <dm...@gmail.com>.
The wrapper sounds like a win (we currently leverage the gradle wrapper to ease the pain).
I’m continuing to tweak the script for j7/android.
> On Jul 11, 2017, at 5:07 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> I'm not demanding the assembly ... It was just the last missing part in the migration to Maven. Most ASF projects provide such convenience binary archives, but an ASF release is the source only anyway.
>
> I admit that your script would do the job.
>
> Another thing worth looking into might be the Maven wrapper, which would eliminate the need to download Maven, as someone not wanting to use Maven would have to install Maven in order to NOT use it afterwards ;-) ... Users using Maven wouldn't use the script.
>
> Chris
Re: maven and distribution bundles with transitive deps:
licensing¬ice
Posted by Christofer Dutz <ch...@c-ware.de>.
I'm not demanding the assembly ... It was just the last missing part in the migration to Maven. Most ASF projects provide such convenience binary archives, but an ASF release is the source only anyway.
I admit that your script would do the job.
Another thing worth looking into might be the Maven wrapper, which would eliminate the need to download Maven, as someone not wanting to use Maven would have to install Maven in order to NOT use it afterwards ;-) ... Users using Maven wouldn't use the script.
Chris
Von meinem Samsung Galaxy Smartphone gesendet.
-------- Ursprüngliche Nachricht --------
Von: Dale LaBossiere <dl...@apache.org>
Datum: 11.07.17 21:29 (GMT+01:00)
An: dev@edgent.apache.org
Betreff: maven and distribution bundles with transitive deps: licensing¬ice
Hi,
Can’t we eliminate releasing binary bundles containing transitive deps and in doing so
eliminate a huge amount of licensing/notice pain for us? But still supply what users need.
As I understand it / ASF policy, the newly generated distribution bundles lack the necessary
license/notice info for transitive deps contained in the bundle.
That’s what the binary-release/LICENSE and licenses/binary-release/* was all about
(included in the gradle generated binary bundles).
Justin indicated we needed to have full copies of license/notice text not just URLs to it
(since the URL may have different text at a later date that doesn’t match that text
applicable for the contained deps).
I see the new distribution stuff also generates edgent-distribution-<ver>.jar.
Its META-INF/DEPENDENCIES has a nice listing of all the transitive deps
license info but it’s (a) just URLs to licenses, (b) lacks any notice info,
and (c) isn’t included in the generated binary release bundle.
Maybe I just don’t understand how this is supposed to work.
Not releasing a binary bundle also eliminates validation/testing of it.
My hope was that viable / better alternative is to provide an easy way for users
themselves to get the Edgent jars and transitive external deps themselves from
maven-central (or any maven repo).
Presumably we'd just need to tell the user something like:
“This command retrieves the Edgent jars and their transitive external dependencies.
The external dependencies come with their own licensing terms that you should review.
A summary of the transitive dependencies and their licenses can be found here
<url to something like the info in the aforementioned META-INF/DEPENDENCIES
and/or binary-release/LICENSING file>.
Continue? <yes|no>”
That’s what get-edgent-jars.sh (https://paste.apache.org/p/GI0n <https://paste.apache.org/p/GI0n>) was working towards.
Is this making sense / compelling / a valid and (sufficiently) user friendly approach?
To clarify for all, this tool (equivalently a binary bundle) is only needed by
Edgent users that don’t use maven/maven-repo enabled app development tools
or that don’t use those tools to generate a standalone “über jar” for their Edgent app
to deploy to their edge device.
— Dale