You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2014/05/14 12:43:12 UTC
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/770
SPARK-1827. LICENSE and NOTICE files need a refresh to contain transitive dependency info
LICENSE and NOTICE policy is explained here:
http://www.apache.org/dev/licensing-howto.html
http://www.apache.org/legal/3party.html
This leads to the following changes.
First, this change enables two extensions to maven-shade-plugin in assembly/ that will try to include and merge all NOTICE and LICENSE files. This can't hurt.
This generates a consolidated NOTICE file that I manually added to NOTICE.
Next, a list of all dependencies and their licenses was generated:
`mvn ... license:aggregate-add-third-party`
to create: `target/generated-sources/license/THIRD-PARTY.txt`
Each dependency is listed with one or more licenses. Determine the most-compatible license for each if there is more than one.
For "unknown" license dependencies, I manually evaluateD their license. Many are actually Apache projects or components of projects covered already. The only non-trivial one was Colt, which has its own (compatible) license.
I ignored Apache-licensed and public domain dependencies as these require no further action (beyond NOTICE above).
BSD and MIT licenses (permissive Category A licenses) are evidently supposed to be mentioned in LICENSE, so I added a section without output from the THIRD-PARTY.txt file appropriately.
Everything else, Category B licenses, are evidently mentioned in NOTICE (?) Same there.
LICENSE contained some license statements for source code that is redistributed. I left this as I think that is the right place to put it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-1827
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/770.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #770
----
commit a764504361ee348d8a304d517584bcb614af4b38
Author: Sean Owen <so...@cloudera.com>
Date: 2014-05-14T10:42:09Z
Add LICENSE and NOTICE info for all transitive dependencies as of 1.0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43117559
E.g. look at how few are mentioned in http://www.us.apache.org/dist/hadoop/common/hadoop-2.4.0/.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/770
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43069220
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14976/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43429692
@srowen Hadoop does distribute binary artifacts that work without a dependency download, so it might be good to let them know about this. Thanks for taking a look at the official policy though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43066220
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43069219
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43117466
I was going to ask the same thing, from what I've seen in other projects. It probably doesn't hurt to keep all of these notices, especially if they're automatically generated, but it may not be necessary. During incubation we were told to put stuff in LICENSE only for libraries we ship in source code form, and in NOTICE only for certain licenses that require mention there (e.g. MPL). I believe MIT, BSD and Apache licensed libraries don't require it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43105049
@srowen thanks a bunch for ton for looking through this. I'll go ahead and merge this and cut a new RC/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43127826
I think the difference is that Spark is distributing its dependencies too in the assembly, whereas I am not sure Hadoop et al. do more than distribute their own artifacts. If that's the not correct, then yeah it's very possible Hadoop doesn't get it right.
I am pretty confident that this is the right thing to do for Spark, and went back to close-read the official word on what goes where. It surprised me a little too. Better safe than sorry, and I think it's buttoned up now to the best of a reasonable person's ability. Thanks for integrating it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43449385
@mateiz That's a good question. I browsed through the Hadoop 2.4.0 binary distribution, and it looks like none of the `hadoop-*` JAR files are 'assembly' JARs -- they all just contain Hadoop-related code. The distro contains _standalone_ third-party JARs in various `lib/` directories though. I would have expected to see similar notices in `NOTICE.txt` and/or `LICENSE.txt` but there is no mention of any of these third-party libraries. Each JAR contains its own `NOTICE` and/or `LICENSE`, I suppose, and that probably technically satisfies the requirement. I still would have expected this to be reproduced, I think. I'll ping it over to those more knowledgeable to see if anyone thinks that needs a change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43117310
Hey @srowen - I noticed that hadoop and a bunch of the other Apache projects have like 3-4 things in the NOTICE file. Are those projects in violation of the "letter of the law" here? It seems like a lot of these dependencies are likely also dependencies of e.g. Hadoop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1827. LICENSE and NOTICE files need a re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/770#issuecomment-43066236
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---