You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@edgent.apache.org by dl...@apache.org on 2016/12/28 20:04:08 UTC
incubator-edgent git commit: [Edgent-329] add Apache RAT task to build
Repository: incubator-edgent
Updated Branches:
refs/heads/master c7cbe49cd -> e47ba5417
[Edgent-329] add Apache RAT task to build
Project: http://git-wip-us.apache.org/repos/asf/incubator-edgent/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-edgent/commit/e47ba541
Tree: http://git-wip-us.apache.org/repos/asf/incubator-edgent/tree/e47ba541
Diff: http://git-wip-us.apache.org/repos/asf/incubator-edgent/diff/e47ba541
Branch: refs/heads/master
Commit: e47ba541758eb4f9e89136b1f6c2a3486315ea84
Parents: c7cbe49
Author: Dale LaBossiere <dl...@us.ibm.com>
Authored: Wed Dec 28 13:57:23 2016 -0500
Committer: Dale LaBossiere <dl...@us.ibm.com>
Committed: Wed Dec 28 13:57:23 2016 -0500
----------------------------------------------------------------------
DEVELOPMENT.md | 37 ++---
build.gradle | 56 ++++++++
gradle/rat.gradle | 136 ++++++++++++++++++
gradle/resources/rat-output-to-html.xsl | 208 +++++++++++++++++++++++++++
4 files changed, 421 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/e47ba541/DEVELOPMENT.md
----------------------------------------------------------------------
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index 6c7ef95..ee0a35c 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -31,7 +31,7 @@ Users of Edgent will need to update their references to the above.
It's recommended that developers of Edgent create a new workspace instead of
reusing their Quarks workspace.
-### Setup
+## Setup
Once you have forked the repository and created your local clone you need to download
these additional development software tools.
@@ -42,7 +42,7 @@ these additional development software tools.
All Edgent runtime development is done using Java 8. JARs for Java 7 and Android
platforms are created as described below.
-### Building a Binary Release Bundle
+## Building a Binary Release Bundle
Building from a source release bundle (lacking a `./gradlew`) requires
performing a one-time bootstrap step using an installed version of gradle:
@@ -61,7 +61,7 @@ be unpacked and used in building applications.
See [Getting Started](https://edgent.apache.org/docs/edgent-getting-started)
for information on using the binary release bundle.
-### Building for Edgent Runtime Development
+## Building for Edgent Runtime Development
The primary build process is using [Gradle](https://gradle.org/),
any pull request is expected to maintain the build success of `clean, assemble, test`.
@@ -98,6 +98,7 @@ unique tasks:
* `reports\tests\overview-summary.html` - JUnit test report
* `reports\coverage\index.html` - Code coverage report
* `release` : Build release bundles in `build/release-edgent`, that includes subsets of the Edgent JARs that run on Java 7 (`build/distributions/java7`) and Android (`build/distributions/android`). By default, SNAPSHOT bundles are created. Specify `-Dedgent.snapshotId=""` to create bundles for a formal release.
+* `rat` : run the Apache Release Analysis Tool (license checking).
* `signAll` : Sign the release bundles in `build/release-edgent` (first run `release`). You will be promoted for your PGP code signing key's ID, the location of the keyring file, and the secret key password. Default response values may be set with environment variables:
* `GPG_ID` - the code signing key's ID (e.g., D0F56CAD)
* `GPG_SECRING` - path to the secret key's keyring file
@@ -106,7 +107,7 @@ The build process has been tested on Linux and macOS.
To build on Windows probably needs some changes, please get involved and contribute them!
-### Continuous Integration
+## Continuous Integration
When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build.
@@ -133,7 +134,7 @@ following statement:
Closing and reopening a pull request will kick off a new build against the pull request.
-### Java 7 and Android
+## Java 7 and Android
Java 7 and Android target platforms are supported through use of
retrolambda to convert Edgent Java8 JARs to Java7 JARs.
@@ -145,7 +146,7 @@ Building a release (`./gradlew release`) produces three sets of JARs under
See [JAVA_SUPPORT.md](JAVA_SUPPORT.md) for which Edgent capabilities / JARs are supported
for each environment.
-#### Adding Edgent Runtime JARs to Java 7 & Android
+### Adding Edgent Runtime JARs to Java 7 & Android
The Gradle tooling uses some Ant tooling to create the Java 7 and Android platform JARs.
@@ -155,13 +156,13 @@ Java 7 Edgent runtime JARs are created using `platform/java7/build.xml`. Adding
Any Java 7 JAR is automatically included in Android unless it is explictly excluded in `platform/android/build.xml`.
-### Test reports
+## Test reports
Running the `reports` target produces two reports:
* `builds/distributions/reports/tests/index.html` - JUnit test report
* `builds/distributions/reports/coverage/index.html` - Code coverage report.
-### Testing the Kafka Connector
+## Testing the Kafka Connector
The kafka connector tests aren't run by default as the connector must
connect to a running Kafka/Zookeeper config.
@@ -184,7 +185,7 @@ cat README
./runkafkasample.sh pub
```
-### Testing the JDBC Connector
+## Testing the JDBC Connector
The JDBC connector tests are written to run against Apache Derby
as the backing dbms and the derby jar needs to be on the classpath.
@@ -212,7 +213,7 @@ cat README
./runjdbcsample.sh reader
```
-### Testing Edgent with Java7
+## Testing Edgent with Java7
All of the standard build system _tasks_ above must be run with
`JAVA_HOME` set to use a Java8 VM.
@@ -238,7 +239,7 @@ $ sh # muck with EVs for Java7 in a subshell
$ ./gradlew test7Reports # generate the JUnit and coverage reports
```
-### Publish to Maven Repository
+## Publish to Maven Repository
Initial support for publishing to a local Maven repository has been added.
Use the following to do the publish.
@@ -254,7 +255,7 @@ names of the JARs in the target-dir / release tgz.
For example: `org.apache.edgent:edgent.api.topology:0.4.0`
-### Code Layout
+## Code Layout
The code is broken into a number of projects and modules within those projects defined by directories under `edgent`.
Each top level directory is a project and contains one or more modules:
@@ -283,7 +284,7 @@ such as an MQTT broker, Apache Kafka, a cloud based IoT service, etc.
* `android` - Code specific to Android.
* `test` - SVT
-### Coding Conventions
+## Coding Conventions
Placeholder: see [EDGENT-23](https://issues.apache.org/jira/browse/EDGENT-23)
@@ -291,8 +292,12 @@ A couple of key items in the mean time:
* Use spaces not hard tabs, indent is 4 spaces
* Don't use wildcard imports
* Don't deliver code with warnings (e.g., unused imports)
+* All source files, scripts, etc must have the standard Apache License header
+ * run the `rat` build task to check license headers
+* Per ASF policy, released source bundles must not contain binaries (e.g., .class, .jar)
+* Per ASF policy, release source and binary bundle LICENSE and NOTICE files must be accurate and up to date, and only bundled 3rd party dependencies whose license meets the ASF licensing requirements can be included.
-### Logging
+## Logging
[SLF4J](http://www.slf4j.org) is used for logging and tracing.
@@ -318,7 +323,7 @@ In general, most code is expected to work on Android (but might not yet) with th
* Functionality aimed at the developer environment, such as console and development provider
* Any JMX related code
-### The ASF / GitHub Integration
+## The ASF / GitHub Integration
The Edgent code is in ASF resident git repositories:
@@ -359,7 +364,7 @@ Notes with the above PR merge directions:
* Use an HTTPS URL unless you have a SSH key setup at GitHub:
- `$ git remote add mirror https://github.com/apache/incubator-edgent.git`
-### Using Eclipse
+## Using Eclipse
The Edgent Git repository contains Eclipse project definitions for the
top-level directories that contain code, such as api, runtime, connectors.
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/e47ba541/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index 5cd08fb..78a0175 100644
--- a/build.gradle
+++ b/build.gradle
@@ -35,6 +35,62 @@ allprojects {
project.version = build_version
}
+apply from: 'gradle/rat.gradle'
+
+rat {
+ excludes = [
+ '*.patch',
+ '**/.classpath',
+ '**/.cache/**',
+ '**/.git/**',
+ '**/.gitignore',
+ '**/.gradle/**',
+ '.gradle-wrapper/**', // historical instead of gradle/wrapper
+ '**/.idea/**',
+ '**/.project',
+ '**/.settings/**',
+ '**/bin/**', // generated by Eclipse builds
+ '**/build/**',
+ '**/META-INF/services/**',
+ '**/README.md',
+ 'externalJars/**', // generated by setupExternalJars for Eclipse use
+ 'gradlew',
+ 'gradlew.bat',
+ 'DEVELOPMENT.md',
+ 'JAVA_SUPPORT.md',
+ 'CONTRIBUTORS', '.mailmap',
+ 'binary-release/apache-notice',
+ 'binary-release/ibm-contrib-notice',
+ 'connectors/jdbc/JdbcStreamsTestDb/**',
+ 'connectors/jdbc/derby.log',
+ 'connectors/mqtt/src/test/keystores/**',
+ 'console/**/js/ext/**/jquery.js', // has header w/MIT license ref
+ 'console/**/js/ext/**/jquery-ui.min.css', // has header w/MIT license ref
+ 'console/**/js/ext/**/jquery-ui.min.js', // has header w/MIT license ref
+ 'console/**/js/ext/**/jquery-ui.structure.min.css', // has header w/MIT license ref
+ 'console/**/js/ext/**/jquery-ui.theme.min.css', // has header w/MIT license ref
+ 'console/**/js/ext/d3.legend.js', // has header w/MIT license ref
+ 'console/**/js/ext/d3.min.js', // no header. included in LICENSE
+ 'console/**/js/ext/sankey_edgent.js', // no header. included in LICENSE
+ 'licenses/d3-sankey.BSD', // OK, BSD 3-clause
+ 'licenses/d3.BSD', // OK, BSD 3-clause
+ 'licenses/binary-release/javax.servlet-api-3.1.0.CDDL-1.0', // OK, only binary form content
+ 'licenses/binary-release/javax.websocket-api-1.0.CDDL-1.1', // OK, only binary form content
+ 'licenses/binary-release/mqttv3-1.0.2.EDL-1.0', // OK, EDL-1.0
+ 'licenses/binary-release/mqttv3-1.1.0.EDL-1.0', // OK, EDL-1.0
+ 'licenses/binary-release/watson-iot-0.1.5.EPL-1.0', // OK, only binary form content
+ 'samples/**/*.properties',
+ 'samples/**/device.cfg',
+ 'scripts/**/*.properties',
+ 'scripts/**/device.cfg',
+ 'scripts/connectors/jdbc/persondata.txt',
+ 'scripts/connectors/kafka/README-kafka',
+ 'test/**/*.properties',
+ 'test/**/device.cfg',
+ 'test/**/*.txt',
+ ]
+}
+
apply plugin: 'java'
jar {
deleteAllActions() // Avoid creating/staging an empty jar for the "root"
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/e47ba541/gradle/rat.gradle
----------------------------------------------------------------------
diff --git a/gradle/rat.gradle b/gradle/rat.gradle
new file mode 100644
index 0000000..2de8999
--- /dev/null
+++ b/gradle/rat.gradle
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// shamelessly copied from https://github.com/apache/samza
+
+import org.gradle.api.Plugin
+import org.gradle.api.Project
+import org.gradle.api.Task
+import org.gradle.api.internal.project.IsolatedAntBuilder
+
+apply plugin: RatPlugin
+
+class RatTask extends DefaultTask {
+ @Input
+ List<String> excludes
+
+ def reportPath = 'build/rat'
+ def stylesheet = 'gradle/resources/rat-output-to-html.xsl'
+ def xmlReport = reportPath + '/rat-report.xml'
+ def htmlReport = reportPath + '/rat-report.html'
+ def textReport = reportPath + '/rat-report.txt'
+
+ def generateTextReport(File reportDir) {
+ def antBuilder = services.get(IsolatedAntBuilder)
+ def ratClasspath = project.configurations.rat
+ antBuilder.withClasspath(ratClasspath).execute {
+ ant.taskdef(resource: 'org/apache/rat/anttasks/antlib.xml')
+ ant.report(format: 'plain', reportFile: textReport) {
+ fileset(dir: ".") {
+ patternset {
+ excludes.each {
+ exclude(name: it)
+ }
+ }
+ }
+ }
+ }
+ }
+
+ def generateXmlReport(File reportDir) {
+ def antBuilder = services.get(IsolatedAntBuilder)
+ def ratClasspath = project.configurations.rat
+ antBuilder.withClasspath(ratClasspath).execute {
+ ant.taskdef(resource: 'org/apache/rat/anttasks/antlib.xml')
+ ant.report(format: 'xml', reportFile: xmlReport) {
+ fileset(dir: ".") {
+ patternset {
+ excludes.each {
+ exclude(name: it)
+ }
+ }
+ }
+ }
+ }
+ }
+
+ def printUnknownFiles() {
+ def ratXml = new XmlParser().parse(xmlReport)
+ def unknownLicenses = 0
+ ratXml.resource.each { resource ->
+ if (resource.'license-approval'.@name[0] == "false") {
+ println('Unknown license: ' + resource.@name)
+ unknownLicenses++
+ }
+ }
+ if (unknownLicenses > 0) {
+ throw new GradleException("Found " + unknownLicenses + " files with " +
+ "unknown licenses.")
+ }
+ }
+
+ def generateHtmlReport() {
+ def antBuilder = services.get(IsolatedAntBuilder)
+ def ratClasspath = project.configurations.rat
+ antBuilder.withClasspath(ratClasspath).execute {
+ ant.xslt(
+ in: xmlReport,
+ style: stylesheet,
+ out: htmlReport,
+ classpath: ratClasspath)
+ }
+ println('Rat report: ' + htmlReport)
+ }
+
+ @TaskAction
+ def rat() {
+ File reportDir = new File(reportPath)
+ if (!reportDir.exists()) {
+ reportDir.mkdirs()
+ }
+ generateXmlReport(reportDir)
+ generateHtmlReport()
+ generateTextReport()
+ printUnknownFiles()
+ }
+}
+
+class RatPlugin implements Plugin<Project> {
+ void apply(Project project) {
+ configureDependencies(project)
+ project.plugins.apply(JavaBasePlugin);
+ Task ratTask = project.task("rat",
+ type: RatTask,
+ group: 'Build',
+ description: 'Runs Apache Rat checks.')
+ project.tasks[JavaBasePlugin.CHECK_TASK_NAME].dependsOn ratTask
+ }
+
+ void configureDependencies(final Project project) {
+ project.configurations {
+ rat
+ }
+ project.repositories {
+ mavenCentral()
+ }
+ project.dependencies {
+ rat 'org.apache.rat:apache-rat-tasks:0.12'
+ }
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/e47ba541/gradle/resources/rat-output-to-html.xsl
----------------------------------------------------------------------
diff --git a/gradle/resources/rat-output-to-html.xsl b/gradle/resources/rat-output-to-html.xsl
new file mode 100644
index 0000000..291ec3e
--- /dev/null
+++ b/gradle/resources/rat-output-to-html.xsl
@@ -0,0 +1,208 @@
+<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
+
+<!--***********************************************************
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ ***********************************************************-->
+
+<!-- This style sheet converts any rat-report.xml file. -->
+
+<!-- shamelessly copied from https://github.com/apache/samza -->
+
+<xsl:template match="/">
+
+ <html>
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
+ <style type="text/css">
+ <!--
+body {margin-top: 0px;font-size: 0.8em;background-color: #F9F7ED;}
+
+h1 {color:red;}
+h2 {color:blue;}
+h3 {color:green;}
+h4 {color:orange;}
+
+/* Table Design */
+
+table,tr,td {text-align:center;font-weight:bold;border:1px solid #000;}
+caption {color:blue;text-align:left;}
+.notes, .binaries, .archives, .standards {width:25%;}
+.notes {background:#D7EDEE;}
+.binaries {background:#D0F2F4;}
+.archives {background:#ABE7E9;}
+.standards {background:#A0F0F4;}
+.licenced, .generated {width:50%;}
+.licenced {background:#C6EBDD;}
+.generated {background:#ABE9D2;}
+.java_note {background:#D6EBC6;}
+.generated_note {background:#C9E7A9;}
+.unknown {width:100%;background:#E92020;}
+.unknown-zero {color:#00CC00;}
+.center{text-align:center;margin:0 auto;}
+-->
+ </style>
+ </head>
+ <body>
+ <xsl:apply-templates/>
+ <xsl:call-template name="generated"/>
+ </body>
+ </html>
+</xsl:template>
+
+<xsl:template match="rat-report">
+
+ <h1>Rat Report</h1>
+ <p>This HTML version (yes, it is!) is generated from the RAT xml reports using Saxon9B. All the outputs required are displayed below, similar to the .txt version.
+ This is obviously a work in progress; and a prettier, easier to read and manage version will be available soon</p>
+<div class="center">
+<table id="rat-reports summary" cellspacing="0" summary="A snapshot summary of this rat report">
+<caption>
+Table 1: A snapshot summary of this rat report.
+</caption>
+ <tr>
+ <td colspan="1" class="notes">Notes: <xsl:value-of select="count(descendant::type[attribute::name="notice"])"/></td>
+ <td colspan="1" class="binaries">Binaries: <xsl:value-of select="count(descendant::type[attribute::name="binary"])"/></td>
+ <td colspan="1" class="archives">Archives: <xsl:value-of select="count(descendant::type[attribute::name="archive"])"/></td>
+ <td colspan="1" class="standards">Standards: <xsl:value-of select="count(descendant::type[attribute::name="standard"])"/></td>
+ </tr>
+ <tr>
+ <td colspan="2" class="licenced">Apache Licensed: <xsl:value-of select="count(descendant::header-type[attribute::name="AL "])"/></td>
+ <td colspan="2" class="generated">Generated Documents: <xsl:value-of select="count(descendant::header-type[attribute::name="GEN "])"/></td>
+ </tr>
+ <tr>
+ <td colspan="2" class="java_note">Note: JavaDocs are generated and so license header is optional</td>
+ <td colspan="2" class="generated_note">Note: Generated files do not require license headers</td>
+ </tr>
+ <tr>
+<xsl:choose>
+ <xsl:when test="count(descendant::header-type[attribute::name="?????"]) > 0">
+ <td colspan="4" class="unknown"><xsl:value-of select="count(descendant::header-type[attribute::name="?????"])"/> Unknown Licenses - or files without a license.</td>
+ </xsl:when>
+ <xsl:otherwise>
+ <td colspan="4" class="unknown-zero"><xsl:value-of select="count(descendant::header-type[attribute::name="?????"])"/> Unknown Licenses - or files without a license.</td>
+ </xsl:otherwise>
+</xsl:choose>
+ </tr>
+</table>
+</div>
+<hr/>
+ <h3>Unapproved Licenses:</h3>
+
+ <xsl:for-each select="descendant::resource[license-approval/@name="false"]">
+ <xsl:text> </xsl:text>
+ <xsl:value-of select="@name"/><br/>
+ <xsl:text>
+</xsl:text>
+</xsl:for-each>
+<hr/>
+
+<h3>Archives:</h3>
+
+<xsl:for-each select="descendant::resource[type/@name="archive"]">
+ + <xsl:value-of select="@name"/>
+ <br/>
+ </xsl:for-each>
+ <hr/>
+
+ <p>
+ Files with Apache License headers will be marked AL<br/>
+ Binary files (which do not require AL headers) will be marked B<br/>
+ Compressed archives will be marked A<br/>
+ Notices, licenses etc will be marked N<br/>
+ </p>
+
+ <xsl:for-each select="descendant::resource">
+ <xsl:choose>
+ <xsl:when test="license-approval/@name="false"">!</xsl:when>
+ <xsl:otherwise><xsl:text> </xsl:text></xsl:otherwise>
+ </xsl:choose>
+ <xsl:choose>
+ <xsl:when test="type/@name="notice"">N </xsl:when>
+ <xsl:when test="type/@name="archive"">A </xsl:when>
+ <xsl:when test="type/@name="binary"">B </xsl:when>
+ <xsl:when test="type/@name="standard""><xsl:value-of select="header-type/@name"/></xsl:when>
+ <xsl:otherwise>!!!!!</xsl:otherwise>
+ </xsl:choose>
+ <xsl:text> </xsl:text>
+ <xsl:value-of select="@name"/><br/>
+ <xsl:text>
+ </xsl:text>
+ </xsl:for-each>
+ <hr/>
+
+ <h3>Printing headers for files without AL header...</h3>
+
+ <xsl:for-each select="descendant::resource[header-type/@name="?????"]">
+
+ <h4><xsl:value-of select="@name"/></h4>
+ <xsl:value-of select="header-sample"/>
+ <hr/>
+</xsl:for-each>
+<br/>
+
+ <!-- <xsl:apply-templates select="resource"/>
+ <xsl:apply-templates select="header-sample"/>
+ <xsl:apply-templates select="header-type"/>
+ <xsl:apply-templates select="license-family"/>
+ <xsl:apply-templates select="license-approval"/>
+ <xsl:apply-templates select="type"/> -->
+
+</xsl:template>
+
+<xsl:template match="resource">
+ <div>
+ <h3>Resource: <xsl:value-of select="@name"/></h3>
+ <xsl:apply-templates/>
+ </div>
+</xsl:template>
+
+<xsl:template match="header-sample">
+ <xsl:if test="normalize-space(.) != ''">
+ <h4>First few lines of non-compliant file</h4>
+ <p>
+ <xsl:value-of select="."/>
+ </p>
+ </xsl:if>
+ <h4>Other Info:</h4>
+</xsl:template>
+
+<xsl:template match="header-type">
+ Header Type: <xsl:value-of select="@name"/>
+ <br/>
+</xsl:template>
+
+<xsl:template match="license-family">
+ License Family: <xsl:value-of select="@name"/>
+ <br/>
+</xsl:template>
+
+<xsl:template match="license-approval">
+ License Approval: <xsl:value-of select="@name"/>
+ <br/>
+</xsl:template>
+
+<xsl:template match="type">
+ Type: <xsl:value-of select="@name"/>
+ <br/>
+</xsl:template>
+
+<xsl:template name="generated">
+</xsl:template>
+</xsl:transform>
\ No newline at end of file