You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@yetus.apache.org by aw...@apache.org on 2019/04/22 16:36:07 UTC
[yetus] branch master updated: YETUS-856. Add markdownlint support
(#47)
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yetus.git
The following commit(s) were added to refs/heads/master by this push:
new 37c8a5f YETUS-856. Add markdownlint support (#47)
37c8a5f is described below
commit 37c8a5f24e006de26d8cb86b1a2d4ccb7939ca5d
Author: Allen Wittenauer <aw...@effectivemachines.com>
AuthorDate: Mon Apr 22 09:36:02 2019 -0700
YETUS-856. Add markdownlint support (#47)
---
.../coprocs.d/README.md => .markdownlint.yaml | 14 +-
README.md | 14 +-
asf-site-src/Gemfile.lock | 32 +-
asf-site-src/source/contribute.html.md | 18 +-
asf-site-src/source/contribute/releases.md | 528 +++++++++++----------
asf-site-src/source/contribute/website.md | 20 +-
.../in-progress/interface-classification.md | 224 ++++-----
.../documentation/in-progress/precommit-admin.md | 34 +-
.../in-progress/precommit-advanced.md | 52 +-
.../documentation/in-progress/precommit-basic.md | 17 +-
.../in-progress/precommit-bugsystems.md | 47 +-
.../in-progress/precommit-buildtools.md | 114 ++---
.../in-progress/precommit-docker-cleanup.md | 9 +-
.../documentation/in-progress/precommit-docker.md | 3 +-
.../in-progress/precommit-patchnames.md | 34 +-
.../documentation/in-progress/precommit-qbt.md | 27 +-
.../documentation/in-progress/precommit-robots.md | 106 ++---
.../in-progress/precommit-testformats.md | 21 +-
.../documentation/in-progress/releasedocmaker.md | 50 +-
.../source/documentation/in-progress/shelldocs.md | 7 +-
.../in-progress/yetus-maven-plugin.md | 15 +-
asf-site-src/source/index.html.md | 1 +
asf-site-src/source/mailinglists.html.md | 2 +
asf-site-src/source/yetus-docker-image.md | 28 +-
.../src/main/shell/coprocs.d/{README.md => README} | 3 +-
.../src/main/shell/test-patch-docker/Dockerfile | 4 +-
.../src/main/shell/test-patch.d/markdownlint.sh | 166 +++++++
27 files changed, 913 insertions(+), 677 deletions(-)
diff --git a/precommit/src/main/shell/coprocs.d/README.md b/.markdownlint.yaml
old mode 100755
new mode 100644
similarity index 75%
copy from precommit/src/main/shell/coprocs.d/README.md
copy to .markdownlint.yaml
index e960cad..d886982
--- a/precommit/src/main/shell/coprocs.d/README.md
+++ b/.markdownlint.yaml
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -14,12 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-#############################
-##
-## bash v4+ coproc routines
-##
-#############################
-
-# bash v3 and lower will treat coproc commands as syntax errors
-# therefore, ALL functions which activate coprocs are located
-# here
\ No newline at end of file
+---
+default: true
+line-length: false
+commands-show-output: false
diff --git a/README.md b/README.md
index 108fd19..88f3974 100644
--- a/README.md
+++ b/README.md
@@ -16,6 +16,7 @@
specific language governing permissions and limitations
under the License.
-->
+
# Apache Yetus
Apache Yetus is a collection of libraries and tools that enable
@@ -27,14 +28,19 @@ Here is a list of the major components:
* [Website source](asf-site-src/)
Holds our documentation, which is presented via [our website](https://yetus.apache.org/).
+
* [Precommit](precommit/)
Precommit provides robust tools to deal with contributions, including applying patches from a variety of project sources and evaluating them against project norms via a system of plugins. See the [precommit overview](asf-site-src/source/documentation/in-progress/precommit-architecture.md) to get started working with precommit.
+
* [Audience Annotations](audience-annotations-component/)
Audience Annotations allows projects to use Java Annotations to delineate public and non-public parts of their APIs. It also provides doclets to generate javadocs filtered by the intended audience. Currently builds with Maven 3.2.0+.
+
* [Shelldocs](shelldocs/)
Shelldocs processes comments on Bash functions for annotations similar to Javadoc. It also includes built-in audience scoping functionality similar to the doclet from Audience Annotations.
+
* [Release Doc Maker](release-doc-maker/)
Release Doc Maker analyzes Jira and Git information to produce Markdown formatted release notes.
+
* [yetus-maven-plugin](yetus-maven-plugin/)
Builds a maven plugin that provides some small utilities for some uncommon maven requirements (such as symlinks) in addition to being mavenized versions of some of the Apache Yetus functionality.
@@ -43,7 +49,8 @@ Builds a maven plugin that provides some small utilities for some uncommon maven
For full instructions on how to build releases and the website, see the [guide to contributing](asf-site-src/source/contribute.html.md) for requirements and instructions.
```bash
-# Launch a Docker container that has all of the project's dependencies and a working build environment
+# Launch a Docker container that has all of the project's dependencies
+# and a working build environment
./start-build-env.sh
# Build the binary tarball, located in yetus-dist/target/artifacts:
@@ -52,7 +59,8 @@ mvn clean install
# Build the binary and source tarballs and sign the content:
mvn clean install -Papache-release
-# Same, but if outside the container and need to let the system know that the OS uses 'gpg2' instead of 'gpg':
+# Same, but if outside the container and need to let the system know
+# that the OS uses 'gpg2' instead of 'gpg':
mvn clean install -Papache-release -Pgpg2
# Build the binary and source tarballs, but skip signing them:
@@ -66,7 +74,7 @@ After executing one or more of the Apache Maven commands, artifacts will be in `
## Container Quickstart
-The project makes available two convenience container images on https://hub.docker.com for both tagged releases and for the master branch. It is highly recommended that casual users use a tagged release so as to not be surprised by incompatible changes that are still rolling through the master branch.
+The project makes available two convenience container images on <https://hub.docker.com> for both tagged releases and for the master branch. It is highly recommended that casual users use a tagged release so as to not be surprised by incompatible changes that are still rolling through the master branch.
### apache/yetus-base
diff --git a/asf-site-src/Gemfile.lock b/asf-site-src/Gemfile.lock
index 293c203..072bb37 100644
--- a/asf-site-src/Gemfile.lock
+++ b/asf-site-src/Gemfile.lock
@@ -1,5 +1,5 @@
GIT
- remote: https://github.com/middleman/middleman-syntax.git
+ remote: git://github.com/middleman/middleman-syntax.git
revision: 0377cc9d2219c2a6ffc2acfd39057f833934af9a
specs:
middleman-syntax (3.1.0)
@@ -7,15 +7,15 @@ GIT
rouge (~> 3.1)
GIT
- remote: https://github.com/vmg/redcarpet.git
- revision: 92a7b3ae2241b862e9bf45e0af3cf53ebdfb0afb
+ remote: git://github.com/vmg/redcarpet.git
+ revision: 94f6e27bdf2395efa555a7c772a3d8b70fb84346
specs:
redcarpet (3.4.0)
GEM
remote: https://rubygems.org/
specs:
- activesupport (4.2.10)
+ activesupport (4.2.11.1)
i18n (~> 0.7)
minitest (~> 5.1)
thread_safe (~> 0.3, >= 0.3.4)
@@ -26,7 +26,7 @@ GEM
rack (>= 1.0.0)
rack-test (>= 0.5.4)
xpath (~> 2.0)
- chunky_png (1.3.10)
+ chunky_png (1.3.11)
coffee-script (2.4.1)
coffee-script-source
execjs
@@ -49,7 +49,7 @@ GEM
erubis (2.7.0)
eventmachine (1.2.7)
execjs (2.7.0)
- ffi (1.9.25)
+ ffi (1.10.0)
haml (5.0.4)
temple (>= 0.8.0)
tilt
@@ -58,7 +58,7 @@ GEM
uber (~> 0.0.14)
http_parser.rb (0.6.0)
i18n (0.7.0)
- json (2.1.0)
+ json (2.2.0)
kramdown (1.17.0)
libv8 (3.16.14.19)
listen (3.0.8)
@@ -98,27 +98,27 @@ GEM
sprockets-sass (~> 1.3.0)
mime-types (3.2.2)
mime-types-data (~> 3.2015)
- mime-types-data (3.2018.0812)
- mini_portile2 (2.3.0)
+ mime-types-data (3.2019.0331)
+ mini_portile2 (2.4.0)
minitest (5.11.3)
multi_json (1.13.1)
- nokogiri (1.8.5)
- mini_portile2 (~> 2.3.0)
+ nokogiri (1.10.2)
+ mini_portile2 (~> 2.4.0)
padrino-helpers (0.12.9)
i18n (~> 0.6, >= 0.6.7)
padrino-support (= 0.12.9)
tilt (>= 1.4.1, < 3)
padrino-support (0.12.9)
activesupport (>= 3.1)
- rack (1.6.10)
+ rack (1.6.11)
rack-livereload (0.3.17)
rack
rack-test (1.1.0)
rack (>= 1.0, < 3)
rake (10.3.1)
rb-fsevent (0.10.3)
- rb-inotify (0.9.10)
- ffi (>= 0.5.0, < 2)
+ rb-inotify (0.10.0)
+ ffi (~> 1.0)
ref (2.0.0)
rouge (3.3.0)
sass (3.4.25)
@@ -132,11 +132,11 @@ GEM
sprockets-sass (1.3.1)
sprockets (~> 2.0)
tilt (~> 1.1)
- temple (0.8.0)
+ temple (0.8.1)
therubyracer (0.12.3)
libv8 (~> 3.16.14.15)
ref
- thor (0.20.0)
+ thor (0.20.3)
thread_safe (0.3.6)
tilt (1.4.1)
tzinfo (1.2.5)
diff --git a/asf-site-src/source/contribute.html.md b/asf-site-src/source/contribute.html.md
index 8f1633f..b8d9801 100644
--- a/asf-site-src/source/contribute.html.md
+++ b/asf-site-src/source/contribute.html.md
@@ -17,23 +17,23 @@
under the License.
-->
-#How To Contribute
+# How To Contribute
## Submitting Changes
We use git as our version control system. To streamline the process of giving proper credit to the contributors when committing patches, we encourage contributors to submit patches generated using git format-patch. This has many benefits:
- * Committers can't forget to attribute proper credit to the contributor
- * The contributors name and email address shows up in git log
- * When viewing Yetus's source code on https://github.com/apache/yetus , the commits from the contributor are linked to their github.com account if it's linked to the same email address they used when generating the git format-patch
+* Committers can't forget to attribute proper credit to the contributor
+* The contributors name and email address shows up in git log
+* When viewing Yetus's source code on <https://github.com/apache/yetus>, the commits from the contributor are linked to their github.com account if it's linked to the same email address they used when generating the git format-patch
Long story short, it makes both the contributors' and committers' lives easier, so please generate your patches using git format-patch.
Here are some instructions on how to generate patches:
- * Ensure that you have all of your change as 1 commit which has the correct commit message - something like `YETUS-1. Update shellcheck plug-in to support bats files`
- * Then run a command like: `git format-patch HEAD^..HEAD --stdout > YETUS-1.00.patch`
- * Upload the YETUS-1.00.patch file to the aforementioned JIRA
+* Ensure that you have all of your change as 1 commit which has the correct commit message - something like `YETUS-1. Update shellcheck plug-in to support bats files`
+* Then run a command like: `git format-patch HEAD^..HEAD --stdout > YETUS-1.00.patch`
+* Upload the YETUS-1.00.patch file to the aforementioned JIRA
The naming of the patch should be in (JIRA).(patch number).patch or, if it needs to apply to a specific branch, (JIRA).(branch name).(patch number).patch format. For example, YETUS-9.00.patch, YETUS-500.02.patch, or YETUS-23.cmake.11.patch. This way, if you need to upload another version of the patch, you should keep the file name the same and JIRA will sort them according to date/time if multiple files have the same name. This feature is also useful to traceback the history of a patch a [...]
@@ -41,5 +41,5 @@ The naming of the patch should be in (JIRA).(patch number).patch or, if it needs
Below are guides meant to give you help accomplishing specific tasks for the project:
- * [Maintaining the Yetus Website](website) - walks through how to view the website locally, update various static and generated pages, and render the html for publishing.
- * [Working with Release Candiates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts.
+* [Maintaining the Yetus Website](website) - walks through how to view the website locally, update various static and generated pages, and render the html for publishing.
+* [Working with Release Candiates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts.
diff --git a/asf-site-src/source/contribute/releases.md b/asf-site-src/source/contribute/releases.md
index 4654848..f95ef57 100644
--- a/asf-site-src/source/contribute/releases.md
+++ b/asf-site-src/source/contribute/releases.md
@@ -1,4 +1,4 @@
-<!---
+<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -17,18 +17,17 @@
under the License.
-->
-<!-- MarkdownTOC levels="1,2" autolink="true" -->
+# Managing a Release
-* [Managing a Release](#managing-a-release)
- * [Dependencies](#dependencies)
- * [Setup](#setup)
- * [Release Candidate\(s\)](#release-candidates)
- * [Verification](#verification)
- * [Cleanup](#cleanup)
+<!-- MarkdownTOC levels="1,2" autolink="true" bullets="-,+,*" -->
-<!-- /MarkdownTOC -->
+- [Dependencies](#dependencies)
+- [Setup](#setup)
+- [Release Candidate\(s\)](#release-candidates)
+- [Verification](#verification)
+- [Cleanup](#cleanup)
-# Managing a Release
+<!-- /MarkdownTOC -->
The Apache Yetus community encourages all committers to help on driving releases. To that end, this section seeks to outline the tools and process you'll use when managing a release. Note that these are our community norms; they do not supersede foundation policy should the two disagree.
@@ -78,11 +77,11 @@ Before attempting to do a release, verify that the documentation, website, etc,
### Ensure Your Public Key is in KEYS
-Like many ASF projects, we provide a single file that downstream folks can use to verify our release artifacts. It's located in the project's distribution area: https://www.apache.org/dist/yetus/KEYS. You can read about this file in the ASF guide to release signing's section [The KEYS File](https://www.apache.org/dist/yetus/KEYS). If your public key is not already included in this file, you will need to add it. You can either follow the instructions in the previously mentioned guide or t [...]
+Like many ASF projects, we provide a single file that downstream folks can use to verify our release artifacts. It's located in the project's distribution area <https://www.apache.org/dist/yetus>. You can read about this file in the [ASF guide to release signing](https://www.apache.org/dev/release-signing) section. If your public key is not already included in [the KEYS file](https://www.apache.org/dist/yetus/KEYS), you will need to add it. You can either follow the instructions in the p [...]
Example commands:
-```
+```bash
$ svn co https://dist.apache.org/repos/dist/release/yetus yetus-dist-release
$ cd yetus-dist-release
$ (gpg --list-sigs <your key name> && gpg --armor --export <your key name>) >> KEYS
@@ -94,32 +93,27 @@ $ svn commit -m "Added myself to KEYS."
Like the rest of our project activity, we'll use an issue in JIRA to track managing the release. You should create this issue and assign it to yourself. As you make your way through the process of creating and running votes on release candidates, this issue will give you a centralized place to collect pointers to your work.
-1. Browse to the ASF JIRA instance's "create issue" page: https://issues.apache.org/jira/secure/CreateIssue!default.jspa
-1. Select "Yetus" for the Project and "Task" for the issue type. Click "Next"
+1. Browse to the ASF JIRA instance's "create issue" page: <https://issues.apache.org/jira/secure/CreateIssue!default.jspa>
+1. Select "Yetus" for the Project and "Task" for the issue type. Click "Next".
1. On the next screen, use a subject line like "Release VERSION", with an appropriate version number. Fill in the following fields and click "Create".
- - The component should be "website and documentation"
- - Affects Version and Fix Version should both be the version you are releasing
- - Assignee should be you
- - Add a description similar to "Generate release candidates as needed to create a VERSION release." with an appropriate version number.
-
-Next, create a shortened link to the JIRA version's release notes. This link should use the ASF link shortener, https://s.apache.org/. To find the appropriate release notes page:
-
-1. Browse to the Yetus JIRA versions page: https://issues.apache.org/jira/browse/YETUS/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel
-1. Click on the Name of the release you are managing
-1. Click on the "Release Notes" button. If it isn't shown, click on the "Summary" button in the left menu
-1. Copy this URL
-1. Browse to the ASF URL shortener: https://s.apache.org/
+ - The component should be "website and documentation"
+ - Affects Version and Fix Version should both be the version you are releasing
+ - Assignee should be you
+ - Add a description similar to "Generate release candidates as needed to create a VERSION release." with an appropriate version number.
+1. Next, create a shortened link to the JIRA version's release notes. This link should use the ASF link shortener, <https://s.apache.org/>. To find the appropriate release notes page:
+ - Browse to the Yetus JIRA versions page: <https://issues.apache.org/jira/browse/YETUS/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel>
+ - Click on the Name of the release you are managing
+ - Click on the "Release Notes" button. If it isn't shown, click on the "Summary" button in the left menu
+ - Copy this URL
+1. Browse to the ASF URL shortener: <https://s.apache.org/>
1. Paste the URL into the "URI" field
1. Set the optional key field to 'yetus-_version_-jira'
-
-For example, on the 0.7.0 release, you would use 'https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318920&version=12334330' for the URI field and 'yetus-0.7.0-jira' for the key.
-
-Finally, you should create a JIRA version that matches the release _following_ the one you are managing. This action is so that folks can continue to work on things that won't make it into the in-progress release while we evaluate candidates.
-
-1. Browse to the ASF JIRA project management page for versions: https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions
-1. Fill in a version one minor version up from the release you're managing. E.g., when managing the 0.7.0 release, fill in 0.3.0.
-1. Set a start date of today.
-1. Click "Add"
+ For example, on the 0.7.0 release, you would use `https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318920&version=12334330` for the URI field and 'yetus-0.7.0-jira' for the key.
+1. Finally, you should create a JIRA version that matches the release _following_ the one you are managing. This action is so that folks can continue to work on things that won't make it into the in-progress release while we evaluate candidates.
+ 1. Browse to the ASF JIRA project management page for versions: <https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions>
+ 1. Fill in a version one minor version up from the release you're managing. E.g., when managing the 0.7.0 release, fill in 0.3.0.
+ 1. Set a start date of today.
+ 1. Click "Add"
### Work in Git
@@ -127,7 +121,7 @@ Once you have an issue to track things, you can create the git branch for stagin
Example commands, presuming the release under management is **0.7.0** and the JIRA issue is **YETUS-XXX**:
-```
+```bash
$ # Ensure master is up to date
$ mvn clean
$ git fetch origin
@@ -163,14 +157,13 @@ yetus-minimaven-plugin/pom.xml
At this point, you should edit the files mentioned above. They must have the version we expect upon a successful release. Search for instances of *VERSION-SNAPSHOT* and replace with *VERSION*; e.g., *0.7.0-SNAPSHOT* should become *0.7.0*:
-```
+```bash
$ perl -pi -e 's,0.7.0-SNAPSHOT,0.7.0,g' $(find . -type f)
```
+After you are done, create a branch-specific patch and then prepare to update the master branch.
- After you are done, create a branch-specific patch and then prepare to update the master branch.
-
-```
+```bash
$ git add -p
$ git commit -m "YETUS-XXX. Stage version 0.7.0."
$ git format-patch --stdout origin/YETUS-XXX > path/to/patches/YETUS-XXX-YETUS-XXX.1.patch
@@ -191,14 +184,13 @@ yetus-minimaven-plugin/pom.xml
Now update these files, but this time you should update them for the next minor version's SNAPSHOT. e.g., *0.7.0-SNAPSHOT* should become *0.8.0-SNAPSHOT*:
-
-```
+```bash
$ perl -pi -e 's,0.7.0-SNAPSHOT,0.8.0-SNAPSHOT,g' $(find . -type f)
```
After you are done, create a patch.
-```
+```bash
$ git add -p
$ git commit -m "YETUS-XXX. bump master version to 0.8.0-SNAPSHOT"
$ git format-patch --stdout origin/master > path/to/patches/YETUS-XXX.1.patch
@@ -210,35 +202,48 @@ Both of these patch files should be uploaded to your release issue for review. P
Depending on how candidate evaluation goes, you may end up performing these steps multiple times. Before you start, you'll need to decide when you want each candidate's vote thread to end. ASF policy requires a minimum voting period of 72 hours (ref [ASF Voting Policy](https://www.apache.org/foundation/voting.html)), so you should ensure enough padding to complete the candidate generation process in time. Ideally, you would plan to post the vote thread on a Friday morning (US time) with [...]
-1. Update JIRA version release date. Browse to the JIRA project version management page (https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions), mark the version as 'Release', and set the release date. Our generated release notes will use this date.
-1. Update your `${HOME}/.m2/settings.xml` file to include the Maven snapshot information as indicated on https://www.apache.org/dev/publishing-maven-artifacts.html
+1. Update JIRA version release date. Browse to the JIRA project version management page <https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions>, mark the version as 'Release', and set the release date. Our generated release notes will use this date.
+1. Update your `${HOME}/.m2/settings.xml` file to include the Maven snapshot information as indicated on <https://www.apache.org/dev/publishing-maven-artifacts.html>
1. Build release artifacts. You should use our convenience script to create the tarballs and markdown documents for a release. Run the following from the release staging branch and inspect the results:
- $ mvn --batch-mode clean deploy -Papache-release
- $ mvn --batch-mode site site:stage
- $ ls -lah yetus-dist/target/artifacts/*
-1. Check out the staging area for release candidates and make a directory for this candidate, somewhere outside of your working directory. Copy the artifacts (**except for the site.tar.gz**) from the previous step into place. For example, when working on RC1 for the 0.7.0 release
+ ```bash
+ $ mvn --batch-mode clean deploy -Papache-release
+ $ mvn --batch-mode site site:stage
+ $ ls -lah yetus-dist/target/artifacts/*
+ ```
+
+1. Check out the staging area for release candidates and make a directory for this candidate, somewhere outside of your working directory. Copy the artifacts from the previous step into place. For example, when working on RC1 for the 0.7.0 release
+
+ ```bash
+ $ svn co `https://dist.apache.org/repos/dist/dev/yetus/` yetus-dist-dev
+ $ cd yetus-dist-dev
+ $ mkdir 0.7.0-RC1
+ $ cd 0.7.0-RC1
+ $ cp path/to/yetus/target/RELEASENOTES.md path/to/yetus/target/CHANGELOG.md path/to/yetus/target/*.tar.gz .
+ ```
- $ svn co https://dist.apache.org/repos/dist/dev/yetus/ yetus-dist-dev
- $ cd yetus-dist-dev
- $ mkdir 0.7.0-RC1
- $ cd 0.7.0-RC1
- $ cp path/to/yetus/target/RELEASENOTES.md path/to/yetus/target/CHANGELOG.md path/to/yetus/target/*.tar.gz .
1. While still in the staging area, sign the artifacts and create the needed checksum files:
- $ for artifact in *; do
- echo ${artifact}
- gpg --use-agent --armor --output "${artifact}".asc --detach-sig "${artifact}"
- gpg --print-mds "${artifact}" >"${artifact}".mds
- shasum -a 512 "${artifact}" >"${artifact}".sha512
- done
+ ```bash
+ $ for artifact in *; do
+ echo ${artifact}
+ gpg --use-agent --armor --output "${artifact}".asc --detach-sig "${artifact}"
+ gpg --print-mds "${artifact}" >"${artifact}".mds
+ shasum -a 512 "${artifact}" >"${artifact}".sha512
+ done
+ ```
+
1. Push the release candidate to staging distribution. This will make the artifacts visible for the vote.
- $ cd ..
- $ svn add 0.7.0-RC1
- $ svn commit -m "stage Apache Yetus 0.7.0-RC1"
-Afterward, the artifacts should be visible via the web under the same URL used when checking out. In the case of 0.7.0-RC1: https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/
+ ```bash
+ $ cd ..
+ $ svn add 0.7.0-RC1
+ $ svn commit -m "stage Apache Yetus 0.7.0-RC1"
+ Afterward, the artifacts should be visible via the web under the same URL used when checking out. In the case of 0.7.0-RC1: <https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/>
+ ```
+
1. Examine staged maven build. Go to the [ASF repository](https://repository.apache.org/) and log in with your asf LDAP credentials. Look for the staging repository with a name that includes "yetus". Clicking on it will give you a link to an "Open" repository. You can examine the structure in the Nexus API while you're logged in. If it looks essentially correct, "Close" the repository. Refreshing and clicking on the repository will give you a link in the Summary tab that other folks can [...]
+1. Create a short link that should point to some online timezone conversion utility that will point to when the vote will end. ASF votes often use timeanddate.com's Event Time Announcer: <http://www.timeanddate.com/worldclock/fixedform.html>.
1. Call a vote on the release candidate. At this point, you have everything you need to call a vote. Your vote thread must contain "[VOTE]" in the subject line, a link to the candidate staging area you created, a source repository commit hash, and voting rules. It should also contain hashes for the artifacts. Here is an example draft for 0.7.0-RC1, update it as appropriate for your release:
Subject: [VOTE] Apache Yetus 0.7.0-RC1
@@ -274,8 +279,6 @@ Afterward, the artifacts should be visible via the web under the same URL used w
[2]: https://www.apache.org/foundation/glossary.html#MajorityApproval
[3]: to find this in your local timezone see:
https://s.apache.org/yetus-0.7.0-rc1-close
-That final short link should point to some online timezone conversion utility. ASF votes often use timeanddate.com's Event Time Announcer: http://www.timeanddate.com/worldclock/fixedform.html.
-
1. Close the vote after the deadline. Once the deadline in the vote thread passes, tally the vote and post a suitable response that changes the subject line to start with "[RESULT]". If the vote failed, ensure there are issues in JIRA for any problems brought up. When they are closed, repeat the steps for creating a release candidate. If the vote passed, proceed to the [Cleanup section](#cleanup)
## Verification
@@ -288,76 +291,89 @@ You will need to download the release candidate files, include the artifacts and
For example, if we use the URL from our exemplar VOTE email, the process would look like this:
- $ wget --recursive --no-parent --quiet 'https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/'
- $ find dist.apache.org/ -type f
-
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.asc
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.sha512
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.mds
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/index.html
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.asc
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.sha512
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.mds
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.asc
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.sha512
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.mds
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.asc
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.sha512
- dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.mds
- dist.apache.org//robots.txt
+```bash
+$ wget --recursive --no-parent --quiet 'https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/'
+$ find dist.apache.org/ -type f
+
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.asc
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.sha512
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.mds
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/index.html
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.asc
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.sha512
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.mds
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.asc
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.sha512
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.mds
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.asc
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.sha512
+dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.mds
+dist.apache.org//robots.txt
+```
Lastly, if you haven't verified a release before, you'll need to download and import the public keys for the project's release managers. The public keys are located in the KEYS file that should have been mentioned in the [VOTE] thread announcement. The specific output of the following commands will vary depending on how many release managers there have been and which keys, if any, you have previously imported.
- $ curl --output KEYS.yetus --silent 'https://www.apache.org/dist/yetus/KEYS'
- $ gpg --import KEYS.yetus
- gpg: key 0D80DB7C: "Sean Busbey (CODE SIGNING KEY) <bu...@apache.org>" not changed
- gpg: Total number processed: 1
- gpg: unchanged: 1
+```bash
+$ curl --output KEYS.yetus --silent 'https://www.apache.org/dist/yetus/KEYS'
+$ gpg --import KEYS.yetus
+gpg: key 0D80DB7C: "Sean Busbey (CODE SIGNING KEY) <bu...@apache.org>" not changed
+gpg: Total number processed: 1
+gpg: unchanged: 1
+```
### ASF required checks
ASF policies require that binding votes on releases be cast only after verifying proper licensing and provenance. For specific details, you should read the [ASF Release Policy's section entitled What Must Every ASF Release Contain?](https://www.apache.org/dev/release.html#what-must-every-release-contain) as well as the informational page [What We Sign](https://www.apache.org/info/verification.html). The following is a non-normative set of guidelines.
-1. You MUST make sure each of the signatures matches. For example, using gpg and taking a fictional source artifact:
+1. You MUST make sure each of the signatures matches. As noted in the informational page [What We Sign](https://www.apache.org/info/verification.html), if you don't have the signer's key in your web of trust the output of the verify command will point this out. You should refer to it for guidance. For example, using gpg and taking a fictional source artifact:
- $ cd dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/
- $ gpg --verify apache-yetus-0.7.0-src.tar.gz.asc apache-yetus-0.7.0-src.tar.gz
- gpg: Signature made Fri Dec 11 11:50:56 2015 CST using RSA key ID 0D80DB7C
- gpg: Good signature from "Sean Busbey (CODE SIGNING KEY) <bu...@apache.org>"
-As noted in the informational page [What We Sign](https://www.apache.org/info/verification.html), if you don't have the signer's key in your web of trust the output of the verify command will point this out. You should refer to it for guidance.
+ ```bash
+ $ cd dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/
+ $ gpg --verify apache-yetus-0.7.0-src.tar.gz.asc apache-yetus-0.7.0-src.tar.gz
+ gpg: Signature made Fri Dec 11 11:50:56 2015 CST using RSA key ID 0D80DB7C
+ gpg: Good signature from "Sean Busbey (CODE SIGNING KEY) <bu...@apache.org>"
+ ```
1. You MUST make sure the provided hashes match the provided artifact.
- $ gpg --print-mds apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_mds
- $ diff apache-yetus-0.7.0-src.tar.gz.mds apache-yetus-0.7.0-src.tar.gz.my_mds
- $ shasum -a 512 apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_sha512
- $ diff apache-yetus-0.7.0-src.tar.gz.sha512 apache-yetus-0.7.0-src.tar.gz.my_sha512
+ ```bash
+ $ gpg --print-mds apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_mds
+ $ diff apache-yetus-0.7.0-src.tar.gz.mds apache-yetus-0.7.0-src.tar.gz.my_mds
+ $ shasum -a 512 apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_sha512
+ $ diff apache-yetus-0.7.0-src.tar.gz.sha512 apache-yetus-0.7.0-src.tar.gz.my_sha512
+ ```
1. You MUST make sure artifacts abide by the ASF Licensing Policy. You should read through [the ASF Licensing Policy](https://www.apache.org/legal/resolved), especially if your vote will be binding. As a quick guide:
- * our software must be under the Apache Software License version 2.0 and this must be noted with a proper LICENSE and NOTICE file in each artifact that can hold them.
- * our source code must meet the ASF policy on proper license notifications. Read the ASF Legal Committee's [Source Header Licensing Guide](https://apache.org/legal/src-headers.html)
- * our LICENSE and NOTICE files must correctly propagate licensing information for bundled products. The [Foundation's Licensing HOWTO Guide](https://www.apache.org/dev/licensing-howto.html) provides guidance on how these files should be maintained.
- * our software must only bundle compatibly licensed products; read [the Licensing Policy's Category A list for compatible licenses](https://www.apache.org/legal/resolved#category-a).
- * our software may only have a runtime dependency on a product with a prohibit license if its use is optional; read [the Licensing Policy's Category X list for prohibited licenses](https://www.apache.org/legal/resolved#category-x) and [the Licensing Policy's explanation of optional runtime dependencies](https://www.apache.org/legal/resolved#optional).
+ - Our software must be under the Apache Software License version 2.0 and this must be noted with a proper `LICENSE` and `NOTICE` file in each artifact that can hold them.
+ - Our source code must meet the ASF policy on proper license notifications. Read the ASF Legal Committee's [Source Header Licensing Guide](https://apache.org/legal/src-headers.html)
+ - Our `LICENSE` and `NOTICE` files must correctly propagate licensing information for bundled products. The [Foundation's Licensing HOWTO Guide](https://www.apache.org/dev/licensing-howto.html) provides guidance on how these files should be maintained.
+ - Our software must only bundle compatibly licensed products; read [the Licensing Policy's Category A list for compatible licenses](https://www.apache.org/legal/resolved#category-a).
+ - Our software may only have a runtime dependency on a product with a prohibit license if its use is optional; read [the Licensing Policy's Category X list for prohibited licenses](https://www.apache.org/legal/resolved#category-x) and [the Licensing Policy's explanation of optional runtime dependencies](https://www.apache.org/legal/resolved#optional).
1. You SHOULD make sure the source release artifact corresponds to the referenced commit hash in the [VOTE] thread. (This ASF policy is currently in DRAFT status.) The release tag is how we'll provide long-term provenance information for our downstream users. Since the release's source code artifact will be the canonical representation of the release we vote on, it is essential that it matches the contents of the version control system's tag. Given our example above, you can check this w [...]
- $ mkdir apache-yetus-0.7.0-src_unpack
- $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
- $ git clone --single-branch --depth=1 --branch YETUS-585 'https://github.com/apache/yetus.git' apache-yetus-0.7.0-RC1-tag
- $ diff -r apache-yetus-0.7.0-RC1-tag apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0
- $ echo $?
- 0
+ ```bash
+ $ mkdir apache-yetus-0.7.0-src_unpack
+ $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
+ $ git clone --single-branch --depth=1 --branch YETUS-585 `https://github.com/apache/yetus.git` apache-yetus-0.7.0-RC1-tag
+ $ diff -r apache-yetus-0.7.0-RC1-tag apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0
+ $ echo $?
+ 0
+ ```
+
1. You MUST make sure any non-source artifacts can be derived from the source artifact. Since the source artifact is the canonical representation of our release, any other artifacts we distribute must be just for the convenience of our downstream users. As such, one must be able to derive them from the source artifact. Currently, you can generate all of the artifacts we distribute for convenience using the same commands used to create the release artifacts.
- $ mkdir apache-yetus-0.7.0-src_unpack
- $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
- $ cd apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0
- $ mvn clean install
-This will create a yetus-dist/target/ directory that contains the tarball binary distribution files.
+ ```bash
+ $ mkdir apache-yetus-0.7.0-src_unpack
+ $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
+ $ cd apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0
+ $ mvn clean install
+ ```
+
+This will create a `yetus-dist/target/` directory that contains the tarball binary distribution files.
### Community recommended checks
@@ -365,133 +381,137 @@ If you've gone through all of the ASF required checks, you'll already have made
1. Test Precommit. The smart-apply-patch and test-patch scripts don't get flexed as a part of the above candidate verification. If you have a downstream project you regularly use, it should suffice to attempt local verification of a contribution. If that project happens to be an ASF project with an example personality, this should be as simple as finding an issue in patch-available status.
- $ cd path/to/my/repo/for/hbase
- $ /some/path/to/the/unpacked/candidate/bin/test-patch --project=hbase HBASE-1772
- ...SNIP...
- -1 overall
-
- | Vote | Subsystem | Runtime | Comment
- ============================================================================
- | 0 | reexec | 0m 0s | Docker mode activated.
- | +1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns.
- | +1 | @\author | 0m 0s | The patch does not contain any @\author
- | | | | tags.
- | +1 | test4tests | 0m 0s | The patch appears to include 2 new or
- | | | | modified test files.
- | +1 | mvninstall | 4m 41s | master passed
- | +1 | compile | 1m 4s | master passed with JDK v1.8.0_72
- | +1 | compile | 0m 57s | master passed with JDK v1.7.0_95
- | +1 | checkstyle | 0m 36s | master passed
- | +1 | mvneclipse | 0m 35s | master passed
- | -1 | findbugs | 1m 6s | hbase-client in master has 19 extant
- | | | | Findbugs warnings.
- | -1 | findbugs | 2m 8s | hbase-server in master has 84 extant
- | | | | Findbugs warnings.
- | -1 | javadoc | 0m 23s | hbase-client in master failed with JDK
- | | | | v1.8.0_72.
- | -1 | javadoc | 0m 34s | hbase-server in master failed with JDK
- | | | | v1.8.0_72.
- | +1 | javadoc | 0m 57s | master passed with JDK v1.7.0_95
- | +1 | mvninstall | 1m 3s | the patch passed
- | +1 | compile | 0m 59s | the patch passed with JDK v1.8.0_72
- | +1 | javac | 0m 59s | the patch passed
- | +1 | compile | 0m 59s | the patch passed with JDK v1.7.0_95
- | +1 | javac | 0m 59s | the patch passed
- | +1 | checkstyle | 0m 32s | the patch passed
- | +1 | mvneclipse | 0m 28s | the patch passed
- | +1 | whitespace | 0m 0s | Patch has no whitespace issues.
- | +1 | hadoopcheck | 4m 28s | Patch does not cause any errors with
- | | | | Hadoop 2.4.1 2.5.2 2.6.0.
- | +1 | findbugs | 3m 37s | the patch passed
- | -1 | javadoc | 0m 24s | hbase-client in the patch failed with
- | | | | JDK v1.8.0_72.
- | -1 | javadoc | 0m 36s | hbase-server in the patch failed with
- | | | | JDK v1.8.0_72.
- | +1 | javadoc | 1m 2s | the patch passed with JDK v1.7.0_95
- | +1 | unit | 1m 23s | hbase-client in the patch passed with
- | | | | JDK v1.8.0_72.
- | -1 | unit | 67m 12s | hbase-server in the patch failed with
- | | | | JDK v1.8.0_72.
- | +1 | unit | 1m 28s | hbase-client in the patch passed with
- | | | | JDK v1.7.0_95.
- | -1 | unit | 66m 16s | hbase-server in the patch failed with
- | | | | JDK v1.7.0_95.
- | +1 | asflicense | 0m 30s | Patch does not generate ASF License
- | | | | warnings.
- | | | 177m 13s |
-
-
- Reason | Tests
- JDK v1.8.0_72 Failed junit tests | hadoop.hbase.client.TestMultiParallel
- JDK v1.7.0_95 Failed junit tests | hadoop.hbase.client.TestMultiParallel
-
-
- || Subsystem || Report/Notes ||
- ============================================================================
- | Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 |
- | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787466/HBASE-1772.patch |
- | JIRA Issue | HBASE-15198 |
- | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
- | uname | Linux 67e02eb9aeea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
- | Build tool | maven |
- | Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
- | git revision | master / 81a6fff |
- | findbugs | v2.0.1 |
- | findbugs | /testptch/patchprocess/branch-findbugs-hbase-client-warnings.html |
- | findbugs | /testptch/patchprocess/branch-findbugs-hbase-server-warnings.html |
- | javadoc | /testptch/patchprocess/branch-javadoc-hbase-client-jdk1.8.0_72.txt |
- | javadoc | /testptch/patchprocess/branch-javadoc-hbase-server-jdk1.8.0_72.txt |
- | javadoc | /testptch/patchprocess/patch-javadoc-hbase-client-jdk1.8.0_72.txt |
- | javadoc | /testptch/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_72.txt |
- | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt |
- | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt |
- | unit test logs | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt |
- | modules | C: hbase-client hbase-server U: . |
- | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
-1. Test Audience Annotations. If you have a downstream project that relies on the audience annotations project, you should be able to install the jars locally and test with the updated verison.
-
- $ mkdir apache-yetus-0.7.0-src_unpack
- $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
- $ cd apache-yetus-0.7.0-src_unpack/yetus-0.7.0
- $ mvn --batch-mode install
- ...SNIP...
- [INFO] ------------------------------------------------------------------------
- [INFO] BUILD SUCCESS
- [INFO] ------------------------------------------------------------------------
- [INFO] Total time: 3.539 s
- [INFO] Finished at: 2016-02-13T02:12:39-06:00
- [INFO] Final Memory: 14M/160M
- [INFO] ------------------------------------------------------------------------
- $ cd path/to/your/project
- $ vim pom.xml # edit version to be e.g. 0.7.0
- $ mvn verify
- ...SNIP...
- [INFO] ------------------------------------------------------------------------
- [INFO] BUILD SUCCESS
- [INFO] ------------------------------------------------------------------------
- [INFO] Total time: 7.539 m
- [INFO] Finished at: 2016-02-13T02:13:39-06:00
- [INFO] Final Memory: 14M/160M
- [INFO] ------------------------------------------------------------------------
+ ```bash
+ $ cd path/to/my/repo/for/hbase
+ $ /some/path/to/the/unpacked/candidate/bin/test-patch --project=hbase HBASE-1772
+ ...SNIP...
+ -1 overall
+
+ | Vote | Subsystem | Runtime | Comment
+ ============================================================================
+ | 0 | reexec | 0m 0s | Docker mode activated.
+ | +1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns.
+ | +1 | @\author | 0m 0s | The patch does not contain any @\author
+ | | | | tags.
+ | +1 | test4tests | 0m 0s | The patch appears to include 2 new or
+ | | | | modified test files.
+ | +1 | mvninstall | 4m 41s | master passed
+ | +1 | compile | 1m 4s | master passed with JDK v1.8.0_72
+ | +1 | compile | 0m 57s | master passed with JDK v1.7.0_95
+ | +1 | checkstyle | 0m 36s | master passed
+ | +1 | mvneclipse | 0m 35s | master passed
+ | -1 | findbugs | 1m 6s | hbase-client in master has 19 extant
+ | | | | Findbugs warnings.
+ | -1 | findbugs | 2m 8s | hbase-server in master has 84 extant
+ | | | | Findbugs warnings.
+ | -1 | javadoc | 0m 23s | hbase-client in master failed with JDK
+ | | | | v1.8.0_72.
+ | -1 | javadoc | 0m 34s | hbase-server in master failed with JDK
+ | | | | v1.8.0_72.
+ | +1 | javadoc | 0m 57s | master passed with JDK v1.7.0_95
+ | +1 | mvninstall | 1m 3s | the patch passed
+ | +1 | compile | 0m 59s | the patch passed with JDK v1.8.0_72
+ | +1 | javac | 0m 59s | the patch passed
+ | +1 | compile | 0m 59s | the patch passed with JDK v1.7.0_95
+ | +1 | javac | 0m 59s | the patch passed
+ | +1 | checkstyle | 0m 32s | the patch passed
+ | +1 | mvneclipse | 0m 28s | the patch passed
+ | +1 | whitespace | 0m 0s | Patch has no whitespace issues.
+ | +1 | hadoopcheck | 4m 28s | Patch does not cause any errors with
+ | | | | Hadoop 2.4.1 2.5.2 2.6.0.
+ | +1 | findbugs | 3m 37s | the patch passed
+ | -1 | javadoc | 0m 24s | hbase-client in the patch failed with
+ | | | | JDK v1.8.0_72.
+ | -1 | javadoc | 0m 36s | hbase-server in the patch failed with
+ | | | | JDK v1.8.0_72.
+ | +1 | javadoc | 1m 2s | the patch passed with JDK v1.7.0_95
+ | +1 | unit | 1m 23s | hbase-client in the patch passed with
+ | | | | JDK v1.8.0_72.
+ | -1 | unit | 67m 12s | hbase-server in the patch failed with
+ | | | | JDK v1.8.0_72.
+ | +1 | unit | 1m 28s | hbase-client in the patch passed with
+ | | | | JDK v1.7.0_95.
+ | -1 | unit | 66m 16s | hbase-server in the patch failed with
+ | | | | JDK v1.7.0_95.
+ | +1 | asflicense | 0m 30s | Patch does not generate ASF License
+ | | | | warnings.
+ | | | 177m 13s |
+
+
+ Reason | Tests
+ JDK v1.8.0_72 Failed junit tests | hadoop.hbase.client.TestMultiParallel
+ JDK v1.7.0_95 Failed junit tests | hadoop.hbase.client.TestMultiParallel
+
+
+ || Subsystem || Report/Notes ||
+ ============================================================================
+ | Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 |
+ | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787466/HBASE-1772.patch |
+ | JIRA Issue | HBASE-15198 |
+ | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
+ | uname | Linux 67e02eb9aeea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
+ | Build tool | maven |
+ | Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
+ | git revision | master / 81a6fff |
+ | findbugs | v2.0.1 |
+ | findbugs | /testptch/patchprocess/branch-findbugs-hbase-client-warnings.html |
+ | findbugs | /testptch/patchprocess/branch-findbugs-hbase-server-warnings.html |
+ | javadoc | /testptch/patchprocess/branch-javadoc-hbase-client-jdk1.8.0_72.txt |
+ | javadoc | /testptch/patchprocess/branch-javadoc-hbase-server-jdk1.8.0_72.txt |
+ | javadoc | /testptch/patchprocess/patch-javadoc-hbase-client-jdk1.8.0_72.txt |
+ | javadoc | /testptch/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_72.txt |
+ | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt |
+ | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt |
+ | unit test logs | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt |
+ | modules | C: hbase-client hbase-server U: . |
+ | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
+ ```
+
+1. Test Audience Annotations. If you have a downstream project that relies on the audience annotations project, you should be able to install the jars locally and test with the updated version.
+
+ ```bash
+ $ mkdir apache-yetus-0.7.0-src_unpack
+ $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz
+ $ cd apache-yetus-0.7.0-src_unpack/yetus-0.7.0
+ $ mvn --batch-mode install
+ ...SNIP...
+ [INFO] ------------------------------------------------------------------------
+ [INFO] BUILD SUCCESS
+ [INFO] ------------------------------------------------------------------------
+ [INFO] Total time: 3.539 s
+ [INFO] Finished at: 2016-02-13T02:12:39-06:00
+ [INFO] Final Memory: 14M/160M
+ [INFO] ------------------------------------------------------------------------
+ $ cd path/to/your/project
+ $ vim pom.xml # edit version to be e.g. 0.7.0
+ $ mvn verify
+ ...SNIP...
+ [INFO] ------------------------------------------------------------------------
+ [INFO] BUILD SUCCESS
+ [INFO] ------------------------------------------------------------------------
+ [INFO] Total time: 7.539 m
+ [INFO] Finished at: 2016-02-13T02:13:39-06:00
+ [INFO] Final Memory: 14M/160M
+ [INFO] ------------------------------------------------------------------------
+ ```
## Cleanup
Once a release candidate obtains majority approval from the PMC, there are several final maintenance tasks you must perform to close out the release.
-1. Create shortcut links to the vote thread (e.g., https://s.apache.org/yetus-0.7.0-rc1-vote) and the result (e.g., https://s.apache.org/yetus-0.7.0-vote-passes) that point to the archives on mail-archives.apache.org. Be aware that it may take several hours for the archive to get the posts that need to be referenced.
-
+1. Create shortcut links to the vote thread (e.g., <https://s.apache.org/yetus-0.7.0-rc1-vote>) and the result (e.g., <https://s.apache.org/yetus-0.7.0-vote-passes>) that point to the archives on mail-archives.apache.org. Be aware that it may take several hours for the archive to get the posts that need to be referenced.
1. Produce a signed release tag. You should create a signed tag and push it to the asf repo. The tag's message should include ASF-shortened links to the vote and results. It should be named 'rel/_version_' so that it will be immutable due to ASF infra's git configuration. Presuming we're working on the 0.7.0 release and the RC1 example above has passed:
$ git config --global user.signingkey <your-key-id> # if you've never configured
$ git tag --sign rel/0.7.0 1e8f4588906a51317207092bd97b35687f2e3fa3
-Example commit message:
+ Example commit message:
YETUS-XXX. tag Apache Yetus 0.7.0 release.
vote thread: https://s.apache.org/yetus-0.7.0-rc1-vote
results: https://s.apache.org/yetus-0.7.0-vote-passes
-Then push:
+ Then push:
$ git push origin rel/0.7.0
1. Add the release to the ASF reporter tool. To make our project reports for the ASF Board easier, you should include the release in the [Apache Committee Report Helper website](https://reporter.apache.org/addrelease.html?yetus). Be sure to use the date release artifacts first were pushed to the distribution area, which should be the same release date as in JIRA. Note that this website is only available to PMC members. If you are not yet in the PMC, please ask them to add the release inf [...]
@@ -503,7 +523,7 @@ Then push:
$ cp path/to/yetus-dist-dev/0.7.0-RC1/* 0.7.0
$ svn add 0.7.0
$ svn commit -m "Publish Apache Yetus 0.7.0"
-It may take up to 24 hours for the artifacts to make their way to the various mirrors. You should not announce the release until after this period.
+ It may take up to 24 hours for the artifacts to make their way to the various mirrors. You should not announce the release until after this period.
1. Remove candidates from the staging area. Once you have moved the artifacts into the distribution area, they no longer need to be in the staging area and should be cleaned up as a courtesy to future release managers.
$ svn co https://dist.apache.org/repos/dist/dev/yetus/ yetus-dist-dev
@@ -534,32 +554,38 @@ It may take up to 24 hours for the artifacts to make their way to the various mi
1. Go to the [ASF repository](https://repository.apache.org/) and click 'Release' to put the RC Maven artifacts into the release repository.
1. Mark JIRA version as released. Browse to the [project version management page for the YETUS JIRA tracker](https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions). Mouse over the version you are managing, click on the gear in the far right and select Release.
1. Delete staging branch. Now that there is an immutable tag for the release, all commits leading up to that release will be maintained by git. Should we need a future maintenance release after this version, we can reestablish the branch based off of the release tag.
-
$ git push origin :YETUS-XXX
1. Update the Mac OS X Homebrew Formula:
- $ vim Formula/yetus.rb
- $ # change URL point to the new version
- $ # update the sha256. e.g., shasum -a 256 bin.gz
- $ # test the formula:
- $ brew install --build-from-source Formula/yetus.rb
+ ```bash
+ $ vim Formula/yetus.rb
+ $ # change URL point to the new version
+ $ # update the sha256. e.g., shasum -a 256 bin.gz
+ $ # test the formula:
+ $ brew install --build-from-source Formula/yetus.rb
+ ```
+
1. Update the documentation in the git master branch for the new release. Remove the oldest release and add the latest.
- $ cd asf-site-src
- $ # Add the release to the releases data file
- $ vim data/versions.yml
- $ vim pom.xml
- $ # add the two stanzas
- $ git add -p
- $ git add asf-site-src/pom.xml
- $ git commit
-Example commit message:
+ ```bash
+ $ cd asf-site-src
+ $ # Add the release to the releases data file
+ $ vim data/versions.yml
+ $ vim pom.xml
+ $ # add the two stanzas
+ $ git add -p
+ $ git add asf-site-src/pom.xml
+ $ git commit
+ ```
- YETUS-XXX. add release 0.7.0.
+ - Example commit message:
- - list in releases
- - remove 0.4.0, add 0.7.0 to pom.xml
+ ```text
+ YETUS-XXX. add release 0.7.0.
+ - list in releases
+ - remove 0.4.0, add 0.7.0 to pom.xml
+ ```
1. You should then post this patch for review. Once you've gotten feedback, it's fine to push the patch to the ASF source repo immediately so long as the updated website is not published.
1. Publish website updates. After the 24 hour window needed for the release artifacts to make their way to the variety of mirrors, you should render the website and publish it using the instructions found in [Maintaining the Yetus Website](../website).
@@ -624,11 +650,11 @@ Example commit message:
--
Meg Smith
Apache Yetus PMC
-If you'd like feedback on the draft, feel free to post it for review on your release issue.
+ If you'd like feedback on the draft, feel free to post it for review on your release issue.
1. Send announcement emails. After the 24 hour window needed for the release artifacts to make their way to the variety of mirrors, you should send the announcement email. The email should come from your apache.org email address and at a minimum should go to the dev@yetus.apache.org and announce@apache.org lists. For details see [the ASF Release Policy section How Should Releases Be Announced?](https://www.apache.org/dev/release.html#release-announcements). Additionally, you may want to [...]
1. Send tweet. Once the message to the ASF-wide announce list has made it to the public archive, you should draft a tweet with a link to the announcement. You should use the ASF link shortener and a descriptive name. For example, the 0.7.0 release could use
Apache Yetus 0.7.0 has been released:
https://s.apache.org/yetus-0.7.0-announce
-This tweet should come from the official [@ApacheYetus](https://twitter.com/ApacheYetus/) account. Currently, only PMC members have access to it. If you are not yet on the PMC, please ask for the PMC to post the tweet once your email is available in the archives.
+ This tweet should come from the official [@ApacheYetus](https://twitter.com/ApacheYetus/) account. Currently, only PMC members have access to it. If you are not yet on the PMC, please ask for the PMC to post the tweet once your email is available in the archives.
diff --git a/asf-site-src/source/contribute/website.md b/asf-site-src/source/contribute/website.md
index c2f24a2..a5e0578 100644
--- a/asf-site-src/source/contribute/website.md
+++ b/asf-site-src/source/contribute/website.md
@@ -19,6 +19,16 @@
# Maintaining the Yetus Website
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Make changes in asf-site-src/source](#make-changes-in-asf-site-srcsource)
+* [Make changes to API Docs](#make-changes-to-api-docs)
+* [Generating the website](#generating-the-website)
+* [Live Development](#live-development)
+* [Publishing the Site](#publishing-the-site)
+
+<!-- /MarkdownTOC -->
+
We use [Middleman](https://middlemanapp.com/) to generate the website content from markdown and other
dynamic templates.If you're interested in digging into how our site makes use of Middleman, or if you run into a problem, you should start
by reading [Middleman's excellent documentation](https://middlemanapp.com/basics/install/).
@@ -27,16 +37,18 @@ by reading [Middleman's excellent documentation](https://middlemanapp.com/basics
NOTE: You MUST have run `mvn install` at least once prior to running `mvn site`.
-The following steps assume you have a working ruby 2.x environment setup:
+The following steps assume you have a working ruby 2.3+ environment setup:
```bash
$ sudo gem install bundler
$ cd asf-site-src
$ bundle install
```
-and a working python 2.x environment for [releasedocmaker](../in-progress/releasedocmaker/).
+
+and a working Python 2.7 environment for [releasedocmaker](../in-progress/releasedocmaker/).
## Make changes in asf-site-src/source
+
Make any changes in the source directory:
```bash
@@ -45,6 +57,7 @@ vi contribute.html.md
```
## Make changes to API Docs
+
Optionally, you can update the generated API docs from other parts of the project. If they have been updated then the middleman build will pick up the changes.
e.g. Precommit changes will be picked up by the Middleman build.
@@ -55,6 +68,7 @@ vi 01-common.sh
```
## Generating the website
+
To generate the static website for Apache Yetus run the following command at the root directory:
```bash
@@ -69,6 +83,7 @@ open asf-site-src/target/site/index.html
```
## Live Development
+
Live development of the site enables automatic reload when changes are saved.
To enable, run the following commands and then open a browser and navigate to
[http://localhost:4567](http://localhost:4567/)
@@ -79,6 +94,7 @@ bundle exec middleman
```
## Publishing the Site
+
Commit the publish directory to the asf-site branch. Presuming we start in a directory that holds your normal Yetus check out:
```bash
diff --git a/asf-site-src/source/documentation/in-progress/interface-classification.md b/asf-site-src/source/documentation/in-progress/interface-classification.md
index b40e8b3..6f1a9dc 100644
--- a/asf-site-src/source/documentation/in-progress/interface-classification.md
+++ b/asf-site-src/source/documentation/in-progress/interface-classification.md
@@ -12,38 +12,45 @@
limitations under the License. See accompanying LICENSE file.
-->
-Apache Yetus Interface Taxonomy: Audience and Stability Classification
-================================================================
+# Apache Yetus Interface Taxonomy: Audience and Stability Classification
-Motivation
-----------
+<!-- MarkdownTOC levels="1,2,3" autolink="true" -->
-The interface taxonomy classification provided by Apache Yetus annotations is for guidance to
-developers and users of interfaces. The classification guides a developer to
-declare the targeted audience or users of an interface and also its stability.
+* [Motivation](#motivation)
+* [Interface Classification](#interface-classification)
+ * [Audience](#audience)
+ * [Private](#private)
+ * [Limited-Private](#limited-private)
+ * [Public](#public)
+ * [Stability](#stability)
+ * [Stable](#stable)
+ * [Evolving](#evolving)
+ * [Deprecated](#deprecated)
+* [How are the Classifications Recorded](#how-are-the-classifications-recorded)
+* [FAQ](#faq)
+
+<!-- /MarkdownTOC -->
+
+# Motivation
+
+The interface taxonomy classification provided by Apache Yetus annotations is for guidance to developers and users of interfaces. The classification guides a developer to declare the targeted audience or users of an interface and also its stability.
* Benefits to the user of an interface: Knows which interfaces to use or not use and their stability.
* Benefits to the developer: to prevent accidental changes of interfaces and
hence accidental impact on users or other components or system. This is
- particularly useful in large systems with many developers who may not all have
- a shared state/history of the project.
+ particularly useful in large systems with many developers who may not all have a shared state/history of the project.
-Interface Classification
-------------------------
+# Interface Classification
Yetus provides the following interface classification, derived from the
[OpenSolaris taxonomy](http://web.archive.org/web/20061013114610/http://opensolaris.org/os/community/arc/policies/interface-taxonomy/)
and, to some extent, from taxonomy used inside Yahoo.
Interfaces have two main attributes: Audience and Stability
-### Audience
+## Audience
-Audience denotes the potential consumers of the interface. While many interfaces
-are internal/private to the implementation, others are public/external interfaces
-and are meant for wider consumption by applications and/or clients. For example,
-POSIX definitions in libc are external, while large parts of the kernel are internal or private interfaces.
-Also, some interfaces are targeted towards other specific subsystems.
+Audience denotes the potential consumers of the interface. While many interfaces are internal/private to the implementation, others are public/external interfaces and are meant for wider consumption by applications and/or clients. For example, POSIX definitions in libc are external, while large parts of the kernel are internal or private interfaces. Also, some interfaces are targeted towards other specific subsystems.
Identifying the audience of an interface helps define the impact of breaking
it. For instance, it might be okay to break the compatibility of an interface
@@ -53,38 +60,36 @@ users depend on.
Yetus uses the following kinds of audience in order of increasing/wider visibility:
-#### Private
+### Private
The interface is for internal use within a project(such as Apache Hadoop)
and should not be used by applications or by other projects. It is subject to
-change at anytime without notice. Most interfaces of a project are Private (also
-referred to as project-private).
+change at anytime without notice. Most interfaces of a project are Private (also referred to as project-private).
-#### Limited-Private
+### Limited-Private
The interface is used by a specified set of projects or systems (typically
closely related projects). Other projects or systems should not use the
interface. Changes to the interface will be communicated/ negotiated with the
-specified projects. For example, in the Apache Hadoop project, some interfaces are
-LimitedPrivate{HDFS, MapReduce} in that they are private to the HDFS and
+specified projects. For example, in the Apache Hadoop project, some interfaces are LimitedPrivate{HDFS, MapReduce} in that they are private to the HDFS and
MapReduce subprojects.
-#### Public
+### Public
The interface is for general use by any application.
-### Stability
+## Stability
Stability denotes how stable an interface is, as in when incompatible changes to
the interface are allowed. Yetus provides the following levels of stability.
-#### Stable
+### Stable
Can evolve while retaining compatibility for minor release boundaries; in other
words, incompatible changes to APIs marked Stable are generally only allowed
at major releases (i.e. at m.0).
-#### Evolving
+### Evolving
Evolving, but incompatible changes are allowed at minor release (i.e. m .x)
@@ -100,111 +105,90 @@ as Not-an-interface is probably more appropriate than "Unstable".
Examples of publicly visible interfaces that are unstable
(i.e. not-an-interface): GUI, CLIs whose output format will change
-#### Deprecated
+### Deprecated
APIs that could potentially be removed in the future and should not be used.
-How are the Classifications Recorded
--------------------------------------
-
+# How are the Classifications Recorded
[//]: # (This section needs improvement. Refer YETUS-458)
-
How should the classification be recorded for the annotated APIs?
* Each interface or class will have the audience and stability recorded using
annotations in org.apache.yetus.classification package.
-
* The javadoc generated by the maven target javadoc:javadoc lists only the public API.
-
-* One can derive the audience of java classes and java interfaces by the
+* One can derive the audience of Java classes and Java interfaces by the
audience of the package in which they are contained. Hence it is useful to
- declare the audience of each java package as public or private (along with the
- private audience variations).
-
-FAQ
----
-
-* Why aren't the java scopes (private, package private and public) good enough?
- * Java's scoping is not very complete. One is often forced to make a class
- public in order for other internal components to use it. It does not have
- friends or sub-package-private like C++.
-
-* But I can easily access a private implementation interface if it is Java public.
- Where is the protection and control?
- * The purpose of this is not providing absolute access control. Its purpose
- is to communicate to users and developers. One can access private
- implementation functions in libc; however if they change the internal
- implementation details, your application will break and you will have
- little sympathy from the folks who are supplying libc. If you use a
- non-public interface you understand the risks.
-
+ declare the audience of each Java package as public or private (along with the private audience variations).
+
+# FAQ
+
+* Why aren't the Java scopes (private, package private and public) good enough?
+ * Java's scoping is not very complete. One is often forced to make a class public in order for other internal components to use it. It does not have friends or sub-package-private like C++.
+* But I can easily access a private implementation interface if it is Java public. Where is the protection and control?
+ * The purpose of this is not providing absolute access control. Its purpose
+ is to communicate to users and developers. One can access private
+ implementation functions in libc; however if they change the internal
+ implementation details, your application will break and you will have
+ little sympathy from the folks who are supplying libc. If you use a
+ non-public interface you understand the risks.
* Why bother declaring the stability of a private interface?
Aren't private interfaces always unstable?
- * Private interfaces are not always unstable. In the cases where they are
- stable they capture internal properties of the system and can communicate
- these properties to its internal users and to developers of the interface.
- * e.g. In HDFS, NN-DN protocol is private but stable and can help
- implement rolling upgrades. It communicates that this interface should
- not be changed in incompatible ways even though it is private.
- * e.g. In HDFS, FSImage stability can help provide more flexible roll backs.
-
-* What is the harm in applications using a private interface that is stable? How
- is it different than a public stable interface?
- * While a private interface marked as stable is targeted to change only at
- major releases, it may break at other times if the providers of that
- interface are willing to changes the internal users of that
- interface. Further, a public stable interface is less likely to break even
- at major releases (even though it is allowed to break compatibility)
- because the impact of the change is larger. If you use a private interface
- (regardless of its stability) you run the risk of incompatibility.
-
-* Why bother with Limited-private? Isn't it giving special treatment to some projects?
- That is not fair.
- * First, most interfaces should be public or private; actually let us state
- it even stronger: make it private unless you really want to expose it to
- public for general use.
- * Limited-private is for interfaces that are not intended for general
- use. They are exposed to related projects that need special hooks. Such a
- classification has a cost to both the supplier and consumer of the limited
- interface. Both will have to work together if ever there is a need to
- break the interface in the future; for example the supplier and the
- consumers will have to work together to get coordinated releases of their
- respective projects. This should not be taken lightly - if you can get
- away with private then do so; if the interface is really for general use
- for all applications then you should consider making it public. But remember
- that making an interface public has huge responsibility. Sometimes
- Limited-private is just right.
- * A good example of a limited-private interface is BlockLocations in the Apache
- Hadoop Project, This is fairly low-level interface that they are willing to
- expose to MR and perhaps HBase. They are likely to change it down the road
- and at that time they will have to get a coordinated effort with the MR
- team to release matching releases. While MR and HDFS are always released
- in sync today, they may change down the road.
- * If you have a limited-private interface with many projects listed then you
- are fooling yourself. It is practically public.
- * It might be worth declaring a special audience classification called
- {YourProjectName}-Private for your closely related projects.
-
-* Can't a private interface be treated as project-private also? For example what is
- the harm in projects in the Apache Hadoop extended ecosystem, having access to
- private classes?
- * Do we want MR accessing class files that are implementation details inside
- HDFS? There used to be many such layer violations in the Apache Hadoop
- project that they have been cleaning up over the last few years. It is highly
- undesirable for such layer violations to creep back in by no separation
- between the major components like HDFS and MR.
-
+ * Private interfaces are not always unstable. In the cases where they are
+ stable they capture internal properties of the system and can communicate
+ these properties to its internal users and to developers of the interface.
+ * e.g. In HDFS, NN-DN protocol is private but stable and can help
+ implement rolling upgrades. It communicates that this interface should
+ not be changed in incompatible ways even though it is private.
+ * e.g. In HDFS, FSImage stability can help provide more flexible roll backs.
+* What is the harm in applications using a private interface that is stable? How is it different than a public stable interface?
+ * While a private interface marked as stable is targeted to change only at
+ major releases, it may break at other times if the providers of that
+ interface are willing to changes the internal users of that
+ interface. Further, a public stable interface is less likely to break even
+ at major releases (even though it is allowed to break compatibility)
+ because the impact of the change is larger. If you use a private interface
+ (regardless of its stability) you run the risk of incompatibility.
+* Why bother with Limited-private? Isn't it giving special treatment to some projects? That is not fair.
+ * First, most interfaces should be public or private; actually let us state
+ it even stronger: make it private unless you really want to expose it to
+ public for general use.
+ * Limited-private is for interfaces that are not intended for general
+ use. They are exposed to related projects that need special hooks. Such a
+ classification has a cost to both the supplier and consumer of the limited
+ interface. Both will have to work together if ever there is a need to
+ break the interface in the future; for example the supplier and the
+ consumers will have to work together to get coordinated releases of their
+ respective projects. This should not be taken lightly - if you can get
+ away with private then do so; if the interface is really for general use
+ for all applications then you should consider making it public. But remember
+ that making an interface public has huge responsibility. Sometimes
+ Limited-private is just right.
+ * A good example of a limited-private interface is BlockLocations in the Apache
+ Hadoop Project, This is fairly low-level interface that they are willing to
+ expose to MR and perhaps HBase. They are likely to change it down the road
+ and at that time they will have to get a coordinated effort with the MR
+ team to release matching releases. While MR and HDFS are always released
+ in sync today, they may change down the road.
+ * If you have a limited-private interface with many projects listed then you are fooling yourself. It is practically public.
+ * It might be worth declaring a special audience classification called
+ {YourProjectName}-Private for your closely related projects.
+* Can't a private interface be treated as project-private also? For example what is the harm in projects in the Apache Hadoop extended ecosystem, having access to private classes?
+ * Do we want MR accessing class files that are implementation details inside
+ HDFS? There used to be many such layer violations in the Apache Hadoop
+ project that they have been cleaning up over the last few years. It is highly
+ undesirable for such layer violations to creep back in by no separation
+ between the major components like HDFS and MR.
* Aren't all public interfaces stable?
- * One may mark a public interface as evolving in its early days. Here one is
- promising to make an effort to make compatible changes but may need to
- break it at minor releases.
- * One example of a public interface that is unstable is where one is
- providing an implementation of a standards-body based interface that is
- still under development. For example, many companies, in an attempt to be
- first to market, have provided implementations of a new NFS protocol even
- when the protocol was not fully completed by IETF. The implementor cannot
- evolve the interface in a fashion that causes least disruption because
- the stability is controlled by the standards body. Hence it is appropriate
- to label the interface as unstable.
+ * One may mark a public interface as evolving in its early days. Here one is
+ promising to make an effort to make compatible changes but may need to
+ break it at minor releases.
+ * One example of a public interface that is unstable is where one is
+ providing an implementation of a standards-body based interface that is
+ still under development. For example, many companies, in an attempt to be
+ first to market, have provided implementations of a new NFS protocol even
+ when the protocol was not fully completed by IETF. The implementor cannot
+ evolve the interface in a fashion that causes least disruption because
+ the stability is controlled by the standards body. Hence it is appropriate
+ to label the interface as unstable.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-admin.md b/asf-site-src/source/documentation/in-progress/precommit-admin.md
index 49f13eb..2ce844a 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-admin.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-admin.md
@@ -17,8 +17,14 @@
under the License.
-->
-Overview
-========
+# Overview
+
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Project-Specifc Builds](#project-specifc-builds)
+* [Jenkins Job Tokens](#jenkins-job-tokens)
+
+<!-- /MarkdownTOC -->
The jenkins-admin script is an automated way to submit JIRA issues to the Apache Yetus precommit testing framework. It works by:
@@ -40,14 +46,13 @@ Additionally, the URL of the Jenkins server is expected to be in the `JENKINS_UR
The very first run of the job should be done with the `--initialize` parameter to create the first `patch_tested.txt` file. Otherwise, the job will fail because a previous version of it cannot be downloaded from previous runs.
-Project-Specifc Builds
-=======================
+# Project-Specifc Builds
New builds are started via buildWithParameters call. Three parameters are added to the URL:
- * token = Jenkins security token (see below)
- * ISSUE\_NUM = JIRA issue number
- * ATTACHMENT\_ID = JIRA attachment id
+* token = Jenkins security token (see below)
+* ISSUE\_NUM = JIRA issue number
+* ATTACHMENT\_ID = JIRA attachment id
By default, the Jenkins job name is expected to be `PreCommit-{project}`, where the project name matches the JIRA project name. Using the JIRA issue YETUS-1 with an attachment number of 2345 would result in the following URL:
@@ -55,7 +60,9 @@ New builds are started via buildWithParameters call. Three parameters are added
The `{JENKINS_URL}` can be overridden on a per project basis using the `--jenkins-url-override` option. This parameter allows for one job on one Jenkins server to direct different projects to different Jenkins servers. For example:
+```bash
jenkins-admin --jenkins-url-override=PROJ1=https://example.com/1 --jenkins-url-override=PROJ2=https://example.com/1
+```
would send all PROJ1 Jenkins jobs to the first URL and all PROJ2 jobs to the second URL. The `--jenkins-url-override` option may be listed as many times as necessary.
@@ -63,21 +70,20 @@ The job name can be overridden via the `--jenkins-project-template` option. For
.../job/PreCommit-YETUS-Build/buildwithParameters?...
-Jenkins Job Tokens
-==================
+# Jenkins Job Tokens
Currently, jenkins-admin supports the usage of Jenkins tokens for authentication via the `--jenkins-token` option. This option provides two ways to do tokens
- * Flat tokens
- * Template tokens
+* Flat tokens
+* Template tokens
Flat tokens are a simple string. For example, `--jenkins-tokens=yetus` would require the `yetus` string to be listed as the token in the Jenkins job configuration that is being requested.
On the other hand, template tokens perform some simple string substitution before being used. This exchange includes:
- * {project} for the JIRA project name
- * {issue} for the JIRA issue number
- * {attachment} for the JIRA issue attachment id
+* {project} for the JIRA project name
+* {issue} for the JIRA issue number
+* {attachment} for the JIRA issue attachment id
For example, if JIRA issue YETUS-1 has an attachment of 2345, then `{project}` becomes `YETUS`, `{issue}` becomes `1`, and `{attachment}` becomes 2345.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
index 98a8fd9..4e5a68e 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
@@ -17,14 +17,25 @@
under the License.
-->
-test-patch
-==========
+# Advanced Precommit
-* [Process Reaper](#test-reaper)
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Process Reaper](#process-reaper)
* [Plug-ins](#plug-ins)
+ * [Common Plug-in Functions](#common-plug-in-functions)
+ * [Plug-in Importation](#plug-in-importation)
+ * [Test Plug-ins](#test-plug-ins)
* [Personalities](#personalities)
+ * [Configuring for Other Projects](#configuring-for-other-projects)
+ * [Global Definitions](#global-definitions)
+ * [Test Determination](#test-determination)
+ * [Module & Profile Determination](#module--profile-determination)
+ * [Enabling Plug-ins](#enabling-plug-ins)
* [Important Variables](#important-variables)
+<!-- /MarkdownTOC -->
+
# Process Reaper
A common problem is the 'stuck' unit test. If bash v4.0 or higher is in use, Apache Yetus may be told to turn on the process reaper functionality. Using the `--reapearmode` option, this feature may be configured to either report and even kill left over processes that match provided regular expressions.
@@ -58,32 +69,31 @@ Similarly, there are other functions that may be defined during the test-patch r
HINT: It is recommended to make the pluginname relatively small, 10 characters at the most. Otherwise, the ASCII output table may be skewed.
* pluginname\_usage
- - executed when the help message is displayed. This is used to display the plug-in specific options for the user.
+ * executed when the help message is displayed. This is used to display the plug-in specific options for the user.
* pluginname\_parse\_args
- - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+ * executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
* pluginname\_initialize
- - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
+ * After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
* pluginname\_docker\_support
- - Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array.
+ * Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array.
* pluginname\_precheck
- - executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch.
+ * executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch.
* pluginname\_patchfile
- - executed prior to the patch being applied but after the git repository is setup. This step is intended to perform tests on the content of the patch itself.
+ * executed prior to the patch being applied but after the git repository is setup. This step is intended to perform tests on the content of the patch itself.
* pluginname\_precompile
- - executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process.
+ * executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process.
* pluginname\_postcompile
- - This step happens after the compile phase.
+ * This step happens after the compile phase.
* pluginname\_rebuild
- - Any non-unit tests that require the source to be rebuilt in a destructive way should be run here.
-
+ * Any non-unit tests that require the source to be rebuilt in a destructive way should be run here.
## Plug-in Importation
@@ -101,29 +111,27 @@ If the `--skip-system-plugins` flag is passed, then only core.d is imported.
Plug-ins geared towards independent tests are registered via:
-
```bash
add_test_type <pluginname>
```
-+ pluginname\_filefilter
- - executed while determining which files trigger which tests. This function should use `add_test pluginname` to add the plug-in to the test list.
+* pluginname\_filefilter
+ * executed while determining which files trigger which tests. This function should use `add_test pluginname` to add the plug-in to the test list.
* pluginname\_compile
- - executed immediately after the actual compilation. This step is intended to be used to verify the results and add extra checking of the compile phase and it's stdout/stderr.
+ * executed immediately after the actual compilation. This step is intended to be used to verify the results and add extra checking of the compile phase and it's stdout/stderr.
* pluginname\_tests
- - executed after the unit tests have completed.
+ * executed after the unit tests have completed.
* pluginname\_clean
- - executed to allow the plugin to remove all files that have been generate by this plugin.
+ * executed to allow the plugin to remove all files that have been generate by this plugin.
* pluginname\_logfilter
- - This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function.
+ * This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function.
* pluginname\_calcdiffs
-
- - This allows for custom log file difference calculation used to determine the before and after views. The default is to use the last column of a colon delimited line of output and perform a diff. If the plug-in does not provide enough context, this may result in error skew. For example, if three lines in a row have "Missing period." as the error, test-patch will not be able to determine exactly which line caused the error. Plug-ins that have this issue will want to use this or po [...]
+ * This allows for custom log file difference calculation used to determine the before and after views. The default is to use the last column of a colon delimited line of output and perform a diff. If the plug-in does not provide enough context, this may result in error skew. For example, if three lines in a row have "Missing period." as the error, test-patch will not be able to determine exactly which line caused the error. Plug-ins that have this issue will want to use this or pote [...]
NOTE: If the plug-in has support for maven, the maven_add_install `pluginname` should be executed. See more information in Custom Maven Tests in the build tool documentation.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md
index 880a388..67a6b3a 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-basic.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md
@@ -17,8 +17,7 @@
under the License.
-->
-test-patch
-==========
+# Basic Precommit
<!-- MarkdownTOC levels="1,2" autolink="true" -->
@@ -114,15 +113,16 @@ Language Support, Licensing, and more:
* [Apache Creadur Rat](http://creadur.apache.org/rat/) entries in build system
* [checkstyle](http://checkstyle.sourceforge.net/) entries in build system (ant and maven only)
* [FindBugs](http://findbugs.sourceforge.net/) entries in build system and 3.x executables
- - NOTE: only one of FindBugs or SpotBugs may be used at a time.
-* [SpotBugs](https://spotbugs.github.io/)) entries in build system and 3.x executables
- - NOTE: only one of FindBugs or SpotBugs may be used at a time.
+ * NOTE: only one of FindBugs or SpotBugs may be used at a time.
* [jshint](https://jshint.com) installed
* [hadolint](https://github.com/hadolint/hadolint) installed
+* [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli) installed
* [Perl::Critic](http://perlcritic.com/) installed
* [pylint](http://www.pylint.org/) installed
* [rubocop](http://batsov.com/rubocop/) installed
* [shellcheck](https://github.com/koalaman/shellcheck) installed, preferably 0.3.6 or higher
+*[SpotBugs](https://spotbugs.github.io/)) entries in build system and 3.x executables
+ * NOTE: only one of FindBugs or SpotBugs may be used at a time.
* [yamllint](https://github.com/adrienverge/yamllint) installed
# First Steps
@@ -162,7 +162,7 @@ $ test-patch --list-plugins
You should see output similar to this:
-```
+```text
BUILDTOOLS:
ant autoconf cmake gradle make maven nobuild
TESTTYPES:
@@ -321,7 +321,6 @@ $ test-patch --plugins=all https://gitlab.com/_a__w_/yetus/merge_requests/3.patc
... will process MR #3 on the \_a\_\_w\_/yetus repo.
-
## Generic URLs
Luckily, `test-patch` supports ways to provide unified diffs via URLs.
@@ -366,7 +365,6 @@ $ test-patch --plugins=all --proclimit=10000
NOTE: The actual implementation of this feature is dependent upon the version of Bash. For bash v4 and higher (most operating systems), the fork bomb protection is generally only used for the build and QA tools. This means Apache Yetus should continue to function. For earlier versions of bash (e.g., OS X), the limit is applied to all of test-patch. If the limit is hit, Apache Yetus will itself likely crash.
-
# MultiJDK
For many projects, it is useful to test Java code against multiple versions of JDKs at the same time. In combination with the `java` plug-in, `test-patch` can do this with the `--multijdkdirs` option:
@@ -383,7 +381,7 @@ NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode. If JA
`test-patch` also has a built-in mode (i.e., no plug-in required) to utilize Docker:
-```bash
+```diff
<<<<<<< HEAD
$ test-patch (other options) --docker
=======
@@ -393,7 +391,6 @@ $ test-patch --docker
This command will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile and other Docker-specific features, see the specific [precommit Docker support](../precommit-docker) page and the [Apache Yetus Docker Hub Images](/yetus-docker-image) page for more information on the convenience Docker images.
-
# In Closing
`test-patch` has many other features and command line options for the basic user. Many of these are self-explanatory. To see the list of options, run `test-patch` without any options or with `--help`.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md b/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md
index d1f3dcc..ea49e5d 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md
@@ -17,8 +17,16 @@
under the License.
-->
-Bug System Support
-==================
+# Bug System Support
+
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Bugzilla Specific](#bugzilla-specific)
+* [GitHub Specific](#github-specific)
+* [GitLab Specific](#gitlab-specific)
+* [JIRA Specific](#jira-specific)
+
+<!-- /MarkdownTOC -->
test-patch has the ability to support multiple bug systems. Bug tools have some extra hooks to fetch patches, line-level reporting, and posting a final report. Every bug system plug-in must have one line in order to be recognized:
@@ -27,28 +35,22 @@ add_bugsystem <pluginname>
```
* pluginname\_locate\_patch
-
- - Given input from the user, download the patch if possible.
+ * Given input from the user, download the patch if possible.
* pluginname\_determine\_branch
-
- - Using any heuristics available, return the branch to process, if possible.
+ * Using any heuristics available, return the branch to process, if possible.
* pluginname\_determine\_issue
-
- - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table.
+ * Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table.
* pluginname\_write\_comment
-
- - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately.
+ * Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately.
* pluginname\_linecomments
-
- - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
+ * This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
* pluginname\_finalreport
-
- - Write the final result table to the bug system.
+ * Write the final result table to the bug system.
# Bugzilla Specific
@@ -60,13 +62,13 @@ $ test-patch (other options) BZ:4
... will pull down Bugzilla ID #4.
-Using the `--bugzilla-base-url` on the command line or BUGZILLA\_BASE\_URL in a project's personality will define the location of the Bugzilla instance. By default, it is https://bz.apache.org/bugzilla .
+Using the `--bugzilla-base-url` on the command line or BUGZILLA\_BASE\_URL in a project's personality will define the location of the Bugzilla instance. By default, it is <https://bz.apache.org/bugzilla>.
# GitHub Specific
GitHub supports the full range of functionality, including putting comments on individual lines. Be aware, however, that `test-patch` will (generally) require that GitHub PRs be fully squashed and rebased (i.e., a single commit) in many circumstances.
-By default, the GitHub plug-in assumes that https://github.com is the base URL for GitHub. Enterprise users may override this with the `--github-base-url` for the normal web user interface and `--github-api-url` for the API URL. Personalities may use GITHUB\_API\_URL and GITHUB\_BASE\_URL.
+By default, the GitHub plug-in assumes that <https://github.com> is the base URL for GitHub. Enterprise users may override this with the `--github-base-url` for the normal web user interface and `--github-api-url` for the API URL. Personalities may use GITHUB\_API\_URL and GITHUB\_BASE\_URL.
The specific repository on GitHub is defined with either `--github-repo` on the command line or GITHUB\_REPO in a personality. It should take the form of "user/repo".
@@ -81,8 +83,8 @@ Both username and password options must be provided.
GitHub pull requests may be directly processed on the command line in two ways:
- * GH:(PR number)
- * GHSHA:(PR SHA1 number)
+* GH:(PR number)
+* GHSHA:(PR SHA1 number)
The GitHub bugsystem plugin will attempt to download the unified diff that the pull request references.
Pull requests that are made off of a specific branch will switch the test repo to that branch, if permitted. If the pull request references a JIRA issue that matches the given JIRA issue regexp in the Subject, the JIRA plug-in will also be invoked as needed.
@@ -91,17 +93,16 @@ Pull requests that are made off of a specific branch will switch the test repo t
GitLab supports the full range of functionality, including putting comments on individual lines. Be aware, however, that `test-patch` will (generally) require that GitLab MRs be fully squashed and rebased (i.e., a single commit) in many circumstances.
-By default, the GitLab plug-in assumes that https://gitlab.com is the base URL for GitHub. Enterprise users may override this with the `--gitlab-base-url` for the normal web user interface and `--gitlab-api-url` for the API URL. Personalities may use GITLAB\_API\_URL and GITLAB\_BASE\_URL.
+By default, the GitLab plug-in assumes that <https://gitlab.com> is the base URL for GitLab. Enterprise users may override this with the `--gitlab-base-url` for the normal web user interface and `--gitlab-api-url` for the API URL. Personalities may use GITLAB\_API\_URL and GITLAB\_BASE\_URL.
The specific repository on GitLab is defined with either `--gitlab-repo` on the command line or GITLAB\_REPO in a personality. It should take the form of "user/repo".
In order to comment on issues or, depending upon the security setup of the repo, authentication credentials. The GitLab plug-in supports tokens via the `--gitlab-token` option or GITLAB\_TOKEN environment variable.
-
GitLab merge requests may be directly processed on the command line in two ways:
- * GL:(MR number)
- * GLSHA:(MR SHA1 number)
+* GL:(MR number)
+* GLSHA:(MR SHA1 number)
The GitLab bugsystem plugin will attempt to download the unified diff that the merge request references.
Merge requests that are made off of a specific branch will switch the test repo to that branch, if permitted. If the merge request references a JIRA issue that matches the given JIRA issue regexp in the Subject, the JIRA plug-in will also be invoked as needed.
@@ -110,7 +111,7 @@ Merge requests that are made off of a specific branch will switch the test repo
JIRA support allows both patch downloads and summary writes. It also supports branch detection-based upon the name of the attached patch file.
-JIRA issues are invoked by matching the command line option to a specific regular expression as given by the `--jira-issue-re` option or via the JIRA\_ISSUE\_RE personality variable. By default, the plug-in uses https://issues.apache.org/jira as the JIRA instance to use. However that may be overwritten via the `--jira-base-url` option or personalities may define via JIRA\_URL.
+JIRA issues are invoked by matching the command line option to a specific regular expression as given by the `--jira-issue-re` option or via the JIRA\_ISSUE\_RE personality variable. By default, the plug-in uses <https://issues.apache.org/jira> as the JIRA instance to use. However that may be overwritten via the `--jira-base-url` option or personalities may define via JIRA\_URL.
In order to write information on the issue, JIRA requires username and password authentication using the `--jira-user`/`--jira-password` options or the JIRA\_USER and JIRA\_PASSWORD variables in a personality.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
index f817073..73911a4 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md
@@ -17,28 +17,38 @@
under the License.
-->
-
-
-Build Tool Support
-===================
+# Build Tool Support
<!-- MarkdownTOC levels="1,2" autolink="true" -->
* [Global Variables](#global-variables)
+ * [BUILDTOOLCWD](#buildtoolcwd)
+ * [UNSUPPORTED\_TEST](#unsupported_test)
* [Required Functions](#required-functions)
+ * [pluginname\_buildfile](#pluginname_buildfile)
+ * [pluginname\_executor](#pluginname_executor)
+ * [pluginname\_modules\_worker](#pluginnamemodulesworker)
+ * [pluginname\_builtin_personality\_modules](#pluginnamebuiltin_personalitymodules)
+ * [pluginname\_builtin_personality\_file\_tests](#pluginnamebuiltin_personality_filetests)
* [Optional Functions](#optional-functions)
-* [Ant Specific](#ant-specific)
- * [Command Arguments](#command-arguments)
- * [Docker Mode](#docker-mode)
+ * [pluginname\_parse\_args](#pluginnameparseargs)
+ * [pluginname\_initialize](#pluginname_initialize)
+ * [pluginname\_reorder\_modules](#pluginnamereordermodules)
+ * [pluginname\_\(test\)\_logfilter](#pluginnametestlogfilter)
+ * [pluginname\_\(test\)_calcdiffs](#pluginnametestcalcdiffs)
+ * [pluginname\_docker\_support](#pluginnamedockersupport)
+* [Apache Ant Specific](#apache-ant-specific)
+ * [Ant Command Arguments](#ant-command-arguments)
+ * [Ant in Docker Mode](#ant-in-docker-mode)
* [autoconf Specific](#autoconf-specific)
- * [Command Arguments](#command-arguments-1)
+ * [autoconf Command Arguments](#autoconf-command-arguments)
* [CMAKE Specific](#cmake-specific)
* [Gradle Specific](#gradle-specific)
* [Make Specific](#make-specific)
-* [Maven Specific](#maven-specific)
- * [Command Arguments](#command-arguments-2)
+* [Apache Maven Specific](#apache-maven-specific)
+ * [Command Arguments](#command-arguments)
* [Per-instance Repositories](#per-instance-repositories)
- * [Docker Mode](#docker-mode-1)
+ * [Maven in Docker Mode](#maven-in-docker-mode)
* [Test Profile](#test-profile)
* [Custom Maven Tests](#custom-maven-tests)
@@ -52,87 +62,79 @@ add_build_tool <pluginname>
# Global Variables
-* BUILDTOOLCWD
-
- - This variable determines where the build tool's command (as returned by pluginname\_executor) should actually execute. It should be one of three values:
-
- * basedir - always execute at the root of the source tree
- * module - switch to the directory as given by the module being processed
- * /(path) - change to the directory as given by this absolute path. If the path does not exist, it will be created.
+## BUILDTOOLCWD
- If /(path) is used, two special substitutions may be made:
+This variable determines where the build tool's command (as returned by pluginname\_executor) should actually execute. It should be one of three values:
+* basedir - always execute at the root of the source tree
+* module - (default) switch to the directory as given by the module being processed
+* /(path) - change to the directory as given by this absolute path. This allows for custom directories to be created and used as necessary. If the path does not exist, it will be created. If /(path) is used, two special substitutions may be made:
* @@@BASEDIR@@@ will be replaced with the root of the source tree
* @@@MODULEDIR@@@ will be replaced with the module name
- This allows for custom directories to be created and used as necessary.
+## UNSUPPORTED\_TEST
-
- The default is module.
-
-* UNSUPPORTED\_TEST
-
- - If pluginname\_modules\_worker is given a test type that is not supported by the build system, set UNSUPPORTED\_TEST=true. If it is supported, set UNSUPPORTED\_TEST=false.
+If pluginname\_modules\_worker is given a test type that is not supported by the build system, set UNSUPPORTED\_TEST=true. If it is supported, set UNSUPPORTED\_TEST=false.
For example, the gradle build tool does not have a standard way to execute checkstyle. So when checkstyle is requested, gradle\_modules\_worker sets UNSUPPORTED\_TEST to true and returns out of the routine.
# Required Functions
-* pluginname\_buildfile
+## pluginname\_buildfile
- - This should be an echo of the file that controls the build system. This is used for module determination. If the build system wishes to disable module determination, it should echo with no args.
+This should be an echo of the file that controls the build system. This is used for module determination. If the build system wishes to disable module determination, it should echo with no args.
-* pluginname\_executor
+## pluginname\_executor
- - This should be an echo of how to run the build tool, any extra arguments, etc.
+This should be an echo of how to run the build tool, any extra arguments, etc.
-* pluginname\_modules\_worker
+## pluginname\_modules\_worker
- - Input is the branch and the test being run. This should call `modules_workers` with the generic parts to run that test on the build system. For example, if it is convention to use 'test' to trigger 'unit' tests, then `modules_workers` should be called with 'test' appended onto its normal parameters.
+Input is the branch and the test being run. This should call `modules_workers` with the generic parts to run that test on the build system. For example, if it is convention to use 'test' to trigger 'unit' tests, then `modules_workers` should be called with 'test' appended onto its normal parameters.
-* pluginname\_builtin_personality\_modules
+## pluginname\_builtin_personality\_modules
- - Default method to determine how to enqueue modules for processing. Note that personalities may override this function. Requires two arguments: repo status and test desired. For example, in a maven build, values may be 'branch' and 'mvninstall'.
+Default method to determine how to enqueue modules for processing. Note that personalities may override this function. Requires two arguments: repo status and test desired. For example, in a maven build, values may be 'branch' and 'mvninstall'.
-* pluginname\_builtin_personality\_file\_tests
+## pluginname\_builtin_personality\_file\_tests
- - Default method to determine which tests to trigger. Note that personalities may override this function. Requires a single argument: the file in which the tests exist.
+Default method to determine which tests to trigger. Note that personalities may override this function. Requires a single argument: the file in which the tests exist.
# Optional Functions
-* pluginname\_parse\_args
+## pluginname\_parse\_args
- - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+Executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
-* pluginname\_initialize
+## pluginname\_initialize
- - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
+After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
-* pluginname\_reorder\_modules
+## pluginname\_reorder\_modules
- - This functions allows the plugin to (re-)order the modules (e.g. based on the output of the maven dependency plugin). When called CHANGED\_MODULES[@] already contains all changed modules. It must be altered to have an effect.
+This functions allows the plugin to (re-)order the modules (e.g. based on the output of the maven dependency plugin). When called CHANGED\_MODULES[@] already contains all changed modules. It must be altered to have an effect.
-* pluginname\_(test)\_logfilter
+## pluginname\_(test)\_logfilter
- - This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. The test plug-in name should be in the (test) part of the function name.
+This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. The test plug-in name should be in the (test) part of the function name.
-* pluginname\_(test)_calcdiffs
+## pluginname\_(test)_calcdiffs
- - Some build tools (e.g., maven) use custom output for certain types of compilations (e.g., java). This allows for custom log file difference calculation used to determine the before and after views.
+Some build tools (e.g., maven) use custom output for certain types of compilations (e.g., java). This allows for custom log file difference calculation used to determine the before and after views.
-* pluginname\_docker\_support
+## pluginname\_docker\_support
- - If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches.
+ If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches.
- **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions. It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command.
+ **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions. It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command.
-# Ant Specific
+# Apache Ant Specific
-## Command Arguments
+## Ant Command Arguments
test-patch always passes -noinput to Ant. This forces ant to be non-interactive.
-## Docker Mode
+## Ant in Docker Mode
In Docker mode, the `${HOME}/.ivy2` directory is shared amongst all invocations.
@@ -140,7 +142,7 @@ In Docker mode, the `${HOME}/.ivy2` directory is shared amongst all invocations.
autoconf requires make to be enabled. autoreconf is always used to rebuild the configure scripte.
-## Command Arguments
+## autoconf Command Arguments
autoconf will always run configure with prefix set to a directory in the patch processing directory. To configure other flags, set the AUTCONF_CONF_FLAGS environment variable.
@@ -158,13 +160,13 @@ In Docker mode, the `${HOME}/.gradle` directory is shared amongst all invocation
No notes.
-# Maven Specific
+# Apache Maven Specific
## Command Arguments
test-patch always passes --batch-mode to maven to force it into non-interactive mode. Additionally, some tests will also force -fae in order to get all of messages/errors during that mode. Some tests are executed with -DskipTests. Additional arguments should be handled via the personality.
-## Per-instance Repositories
+## Per-instance Repositories
Under many common configurations, maven (as of 3.3.3 and lower) may not properly handle being executed by multiple processes simultaneously, especially given that some tests require the `mvn install` command to be used.
@@ -174,7 +176,7 @@ By default, `test-patch` uses `${HOME}/yetus-m2` as the base directory to store
The location of the `settings.xml` may be changed via the `--mvn-settings` option.
-## Docker Mode
+## Maven in Docker Mode
In Docker mode, `${HOME}/.m2` is shared amongst all invocations. If `--mvn-custom-repos` is used, all of `--mvn-custom-repos-dir` is shared with all invocations. The per-instance directory will be calculated and configured after Docker has launched.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md b/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md
index 9df7524..9add204 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md
@@ -17,8 +17,12 @@
under the License.
-->
-docker-cleanup
-==============
+# docker-cleanup
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Usage](#usage)
+
+<!-- /MarkdownTOC -->
`docker-cleanup` is a command to perform precommit's Docker cleanup functionality outside of patch and build testing.
@@ -40,7 +44,6 @@ Deleted: sha256:1a96c79a0a9ab538c6c7765dc908eca3e689270b778d6ae2add558e89792a7d8
```
-
`docker-cleanup` also supports the `--sentinel` mode to kill and remove stale running containers:
```bash
diff --git a/asf-site-src/source/documentation/in-progress/precommit-docker.md b/asf-site-src/source/documentation/in-progress/precommit-docker.md
index 8bd344c..1930710 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-docker.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-docker.md
@@ -17,8 +17,7 @@
under the License.
-->
-test-patch Docker Support
-=========================
+# test-patch Docker Support
<!-- MarkdownTOC levels="1,2" autolink="true" -->
diff --git a/asf-site-src/source/documentation/in-progress/precommit-patchnames.md b/asf-site-src/source/documentation/in-progress/precommit-patchnames.md
index ce89b92..51b941c 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-patchnames.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-patchnames.md
@@ -17,29 +17,35 @@
under the License.
-->
-We use Apache Yetus to process your patch. It supports the following patterns and
-procedures for patch file names:
+# Patch File Naming
+
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [JIRA](#jira)
+* [Github](#github)
+
+<!-- /MarkdownTOC -->
+
+We use Apache Yetus to process your patch. It supports the following patterns and procedures for patch file names:
## JIRA
If JIRA support is configured, attach the patch to the given ISSUE and
click 'Submit Patch'. The patch file should be named one of:
- * ISSUE.patch
- * ISSUE.###.patch
- * ISSUE.branch.###.patch
- * ISSUE-branch-###.patch
- * ISSUE.###.branch.patch
- * ISSUE-branch.###.patch
+* ISSUE.patch
+* ISSUE.###.patch
+* ISSUE.branch.###.patch
+* ISSUE-branch-###.patch
+* ISSUE.###.branch.patch
+* ISSUE-branch.###.patch
The meaning of the subpart are the following:
- * ISSUE: JIRA key (e.g. YETUS-1)
- * branch: either the name of a branch or a git hash prefixed with **git** (e.g. branch-2.8 or git8e55427b35)
- * ###: revision of the patch (e.g. 00 or 01)
+* ISSUE: JIRA key (e.g. YETUS-1)
+* branch: either the name of a branch or a git hash prefixed with **git** (e.g. branch-2.8 or git8e55427b35)
+* ###: revision of the patch (e.g. 00 or 01)
## Github
-If Github support is also configured, a comment that contains a link to a Pull Request's
-patch file (e.g., https://github.com/user/repo/pull/1.patch) will pull the patch from
-the given Github PR.
+If Github support is also configured, a comment that contains a link to a Pull Request's patch file (e.g., <https://github.com/user/repo/pull/1.patch>) will pull the patch from the given Github PR.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-qbt.md b/asf-site-src/source/documentation/in-progress/precommit-qbt.md
index 91d1749..665bd8c 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-qbt.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-qbt.md
@@ -17,13 +17,19 @@
under the License.
-->
-qbt
-===
+# qbt
-`qbt` is a command to execute test-patch without a patch. It uses
-the same plug-ins and the same options as test-patch. The only
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [Reporting](#reporting)
+* [Archiving](#archiving)
+
+<!-- /MarkdownTOC -->
+
+`qbt` is a command to execute `test-patch` without a patch. It uses
+the same plug-ins and the same options as `test-patch`. The only
difference is that no patch file, location, etc should be supplied.
-It is meant to be a way to easily get test-patch's output on your
+It is meant to be a way to easily get `test-patch`'s output on your
current source tree. It is suitable to be run as a regularly
scheduled build as part of your overall development strategy.
@@ -34,7 +40,7 @@ When using an automation tool, it may be useful to use the
file. This can then be used with systems like Jenkin's
email-ext plug-in to send the output as an emailed report:
-```
+```text
${FILE,path="<report-file-path>"}
```
@@ -42,7 +48,7 @@ For something a bit more structured, there is also the `--html-report-file`
option. Using this output, again with Jenkins' email-ext plug-in, it is
possible to build some very nice looking output that is easily customized:
-```
+```html
<html>
<head>
<style>
@@ -63,12 +69,7 @@ ${FILE,path="<report-file-path>"}
</body></html>
```
-If your mailing lists do not allow HTML-formatted email, then the `--brief-report-file`
-provides a solution. This option creates a very plain, reduced content text file
-suitable for email. It contains just the barebones information needed to get
-information on failures: what voted -1, what tests failed, what subsystems are long
-running (configurable with the `--brief-report-long` opton), and a list of any
-attached log files.
+If your mailing lists do not allow HTML-formatted email, then the `--brief-report-file` provides a solution. This option creates a very plain, reduced content text file suitable for email. It contains just the barebones information needed to get information on failures: what voted -1, what tests failed, what subsystems are long running (configurable with the `--brief-report-long` opton), and a list of any attached log files.
NOTE: Be aware that ASF mailing lists do not allow HTML formatted email.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-robots.md b/asf-site-src/source/documentation/in-progress/precommit-robots.md
index 6fb44ab..8e73c1d 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-robots.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-robots.md
@@ -17,9 +17,7 @@
under the License.
-->
-
-Robots: Continuous Integration Support
-======================================
+# Robots: Continuous Integration Support
<!-- MarkdownTOC levels="1,2" autolink="true" -->
@@ -35,22 +33,20 @@ Robots: Continuous Integration Support
<!-- /MarkdownTOC -->
-Intro
-=====
+# Intro
`test-patch` works hand-in-hand with various CI and other automated build systems. `test-patch` will attempt to auto-determine if it is running under such a system and change its defaults to match known configuration parameters automatically. When robots are activated, there is generally some additional/changed behavior:
- * display extra information in the footer
- * change log entries from file names to URLs
- * activate `--resetrepo` to keep the directory structure clean
- * enable the running of unit tests and run them in parallel
- * if possible, write comments to bug systems
- * attempt to determine the build tool in use
- * activate Docker maintenance when `--docker` is passed
- * attempt to determine whether this is a full build (`qbt`) or testing a patch/merge request/pull request.
+* display extra information in the footer
+* change log entries from file names to URLs
+* activate `--resetrepo` to keep the directory structure clean
+* enable the running of unit tests and run them in parallel
+* if possible, write comments to bug systems
+* attempt to determine the build tool in use
+* activate Docker maintenance when `--docker` is passed
+* attempt to determine whether this is a full build (`qbt`) or testing a patch/merge request/pull request.
-Azure Pipelines
-===============
+# Azure Pipelines
NOTE: Azure Pipelines support is not stable and should be viewed as experimental, at best.
@@ -60,8 +56,7 @@ Azure Pipelines support has only been tested on the Ubuntu VM with GitHub as the
As of this writing, Azure Pipelines has moved to a custom moby build for the 'docker' executable. As a result, `--docker` is not supported.
-Circle CI
-=========
+# Circle CI
TRIGGER: ${CIRCLECI}=true
@@ -70,10 +65,11 @@ Circle CI support in `test-patch` is limited to github.com.
To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `.circleci/config.yaml` file, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile:
```yaml
+---
jobs:
build:
docker:
- - image: apache/yetus:0.9.0
+ - image: apache/yetus:0.10.0
environment:
JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
@@ -84,6 +80,7 @@ jobs:
Artifacts need some special handling. In order to get links, the storage of artifacts must be 'primed' prior to launching test-patch and then again to actually store the content. Additionally, the location needs to be handled set on the command line. In practice, this configuration looks similar to this:
```yaml
+---
jobs:
build:
steps:
@@ -100,28 +97,29 @@ jobs:
path: /tmp/yetus-out
```
-See also
- * See also the source tree's `.circleci/config.yaml` for some tips and tricks.
+See also:
+
+* Apache Yetus' source tree [.circleci/config.yaml](https://github.com/apache/yetus/blob/master/.circleci/config.yml) for some tips and tricks.
-Gitlab CI
-=========
+# Gitlab CI
TRIGGER: ${GITLAB_CI}=true
Artifacts, patch logs, etc are configured to go to a yetus-out directory in the source tree after completion. Adding this stanza to your `.gitlab-ci.yml` file will upload and store those components for a week in Gitlab CI's artifact retrieval system:
```yaml
+---
artifacts:
expire_in: 1 week
when: always
paths:
- yetus-out/
-
```
-To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `.gitlab-ci.yml` file, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile:
+To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `.gitlab-ci.yml` file, substituting the tag for the version of Apache Yetus that should be used and replacing the `JAVA_HOME` with the appropriate version as bundled mentioned in the Dockerfile:
```yaml
+---
job:
image: apache/yetus:0.9.0
allow_failure: true
@@ -131,20 +129,21 @@ job:
...
```
-See also
- * See also the source tree's `.gitlab-ci.yml` for some tips and tricks.
+See also:
-Jenkins
-=======
+* Apache Yetus' source tree [.gitlab-ci.yml](https://github.com/apache/yetus/blob/master/.gitlab-ci.yml) for some tips and tricks.
+
+# Jenkins
TRIGGER: ${JENKINS_URL}=(anything) , ${EXECUTOR_NUMBER}=(anything)
-Jenkins is extremely open-ended and, given multiple executors, does not run workflows in isolation. As a result, many more configuration options generally need to be configured as it is not safe or may be suprising to users for test-patch to autodetermine some settings. By default, Jenkins will trigger a full build.
+Jenkins is extremely open-ended and, given multiple executors, does not run workflows in isolation. As a result, many more configuration options generally need to be configured as it is not safe or may be surprising to users for test-patch to auto-determine some settings. By default, Jenkins will trigger a full build.
There is some support for a few well known environment variables:
- * `${CHANGE_URL}` or `${ghprbPullLink}` will set the patch location as well as trigger some extra handling if 'github' or 'gitlab' appear in the string.
- * `${GIT_URL}` will trigger the same extra handling if 'github' or 'gitlab' appear in the string.
- * If `${ghprbPullId}` is set, then test-patch will configure itself for a Github-style PR.
+
+* `${CHANGE_URL}` or `${ghprbPullLink}` will set the patch location as well as trigger some extra handling if 'github' or 'gitlab' appear in the string.
+* `${GIT_URL}` will trigger the same extra handling if 'github' or 'gitlab' appear in the string.
+* If `${ghprbPullId}` is set, then test-patch will configure itself for a Github-style PR.
To use the pre-built Apache Yetus Docker image from docker hub as the build environment, use the following snippet in the `Jenkinsfile`, substituting the tag for the version of Apache Yetus that should be used and replacing the JAVA_HOME with the appropriate version as bundled mentioned in the Dockerfile:
@@ -165,7 +164,7 @@ pipeline {
```
- Experience has shown that certain Jenkins + Java + OS combinations have problems sending signals to child processes. In the case of Apache Yetus, this may result in aborted or workflows that timeout not being properly killed. `test-patch` will write two files in the patch directory that may be helpful to combat this situation if it applies to your particular configuration. `pidfile.txt` contains the master `test-patch` process id and `cidfile.txt` contains the docker container id. T [...]
+Experience has shown that certain Jenkins + Java + OS combinations have problems sending signals to child processes. In the case of Apache Yetus, this may result in aborted or workflows that timeout not being properly killed. `test-patch` will write two files in the patch directory that may be helpful to combat this situation if it applies to your particular configuration. `pidfile.txt` contains the master `test-patch` process id and `cidfile.txt` contains the docker container id. Th [...]
```groovy
post {
@@ -188,17 +187,15 @@ pipeline {
}
```
+See also:
+* Apache Yetus' source tree [Jenkinsfile](https://github.com/apache/yetus/blob/master/Jenkinsfile) for some tips and tricks.
+* [precommit-admin](precommit-admin), for special utilities built for Jenkins.
+* [GitHub Branch Source Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin)
+* [GitHub Pull Request Builder Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin)
+* `https://{your local server}/env-vars.html/`
-See also
- * See also the source tree's `Jenkinsfile` for some tips and tricks.
- * [precommit-admin](precommit-admin), for special utilities built for Jenkins.
- * [GitHub Branch Source Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin)
- * [GitHub Pull Request Builder Plugin](https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin)
- * https://{your local server}/env-vars.html/
-
-Semaphore CI
-=========
+# Semaphore CI
NOTE: Semaphore CI support is not stable and should be viewed as experimental, at best.
@@ -206,10 +203,14 @@ TRIGGER: ${CI}=true and ${SEMAPHORE}=true
Semaphore CI requires that `checkout --use-cache` has been used prior to trigging test-patch. It is HIGHLY recommended to use a helper script checked into the repository to control precommit options to avoid problems with Semaphore CI's parsing of long lines in the YAML file.
- The GitHub repo and the Pull Request in use are automatically detected. However, some personalities may override the auto-detected Github repository information. It may be necessary to manually configure it in your `.semaphore.yml` file.
+The GitHub repo and the Pull Request in use are automatically detected. However, some personalities may override the auto-detected Github repository information. It may be necessary to manually configure it in your `semaphore.yml` file.
-Travis CI
-=========
+See also:
+
+* Apache Yetus' source tree [semaphore.yml](https://github.com/apache/yetus/blob/master/.semaphore/semaphore.yml) for some tips and tricks.
+* Apache Yetus' helper script [semaphore-build.sh](https://github.com/apache/yetus/blob/master/.semaphore/semaphore-build.sh)
+
+# Travis CI
TRIGGER: ${TRAVIS}=true
@@ -219,16 +220,15 @@ If `${ARTIFACTS_PATH}` is configured, then `--patch-dir` is set to the first lis
Personalities will override the auto-detected Github repository information. It may be necessary to manually configure it in your `.travis.yml` file.
-As of this writing, it is not possible to make the Travis CI build environment use the Apache Yetus pre-built docker images without using ` docker run` in the before_install phase. Therefore, using the image is the same as described in the [Apache Yetus Docker Hub Images](/yetus-docker-image) page.
+As of this writing, it is not possible to make the Travis CI build environment use the Apache Yetus pre-built docker images without using `docker run` in the before_install phase. Therefore, using the image is the same as described in the [Apache Yetus Docker Hub Images](/yetus-docker-image) page.
-See also
- * See also the source tree's `.travis.yml` for some tips and tricks.
+See also:
-Manual Configuration
-====================
+* Apache Yetus' source tree [.travis.yml](https://github.com/apache/yetus/blob/master/.travis.yml) for some tips and tricks.
-For automated systems that are not directly supported, `--robot` tells `test-patch` that this is an automated system. This will trigger many of the above settings.
+# Manual Configuration
+For automated systems that are not directly supported, `--robot` tells `test-patch` that this is an automated system. This will trigger many of the above settings.
The `--build-url` option is also useful when running in `--robot` mode so that emails and such
have a location to look at the output artifacts:
@@ -245,8 +245,6 @@ $ test-patch --robot --instance=1
If `--robot` is specified without an instance, a random number is generated and used.
-
-Sentinel Mode
-=============
+# Sentinel Mode
If stuck Docker containers are a problem, a more aggressive robot may be enabled with the `--sentinel` option. This option enables killing containers that have been running for over 24 hours as well.
\ No newline at end of file
diff --git a/asf-site-src/source/documentation/in-progress/precommit-testformats.md b/asf-site-src/source/documentation/in-progress/precommit-testformats.md
index fcbfc36..6eda542 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-testformats.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-testformats.md
@@ -17,8 +17,15 @@
under the License.
-->
-Test Format Support
-==================
+# Test Format Support
+
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [pluginname\_process\_tests](#pluginnameprocesstests)
+* [pluginname\_finalize\_results](#pluginnamefinalizeresults)
+* [Example](#example)
+
+<!-- /MarkdownTOC -->
test-patch has the ability to support multiple test formats. Test formats have some extra hooks to process the output of test tools and write the results to some tables. Every test format plug-in must have one line in order to be recognized:
@@ -28,12 +35,14 @@ add_test_format <pluginname>
Test format plugins can provide following two methods, which will be called by test-patch if defined.
-* pluginname\_process\_tests
+## pluginname\_process\_tests
+
+Given a path to the log file and tested module name, parse that file and store the test result into global variables and/or files.
- - Given a path to the log file and tested module name, parse that file and store the test result into global variables and/or files.
+## pluginname\_finalize\_results
-* pluginname\_finalize\_results
+Using the results stored by pluginname\_process\_tests, write them to the test result table and/or the footer table for reporting.
- - Using the results stored by pluginname\_process\_tests, write them to the test result table and/or the footer table for reporting.
+# Example
For an example of how to write a test-format plugin, you can look at [junit plugin](https://github.com/apache/yetus/blob/master/precommit/test-patch.d/junit.sh) bundled in Apache Yetus.
\ No newline at end of file
diff --git a/asf-site-src/source/documentation/in-progress/releasedocmaker.md b/asf-site-src/source/documentation/in-progress/releasedocmaker.md
index 13b3eb0..ce504d0 100644
--- a/asf-site-src/source/documentation/in-progress/releasedocmaker.md
+++ b/asf-site-src/source/documentation/in-progress/releasedocmaker.md
@@ -17,18 +17,21 @@
under the License.
-->
-releasedocmaker
-===============
+# releasedocmaker
<!-- MarkdownTOC levels="1,2" autolink="true" -->
* [Purpose](#purpose)
* [Requirements](#requirements)
* [Basic Usage](#basic-usage)
+* [Authentication](#authentication)
* [Changing the Header](#changing-the-header)
+* [Versioned Files and Directories](#versioned-files-and-directories)
* [Multiple Versions](#multiple-versions)
* [Unreleased Dates](#unreleased-dates)
* [Sorted Output](#sorted-output)
+ * [Resolution Date-base Sort](#resolution-date-base-sort)
+ * [Issue Number-based Sort](#issue-number-based-sort)
* [Backward Incompatible Changes](#backward-incompatible-changes)
* [Lint Mode](#lint-mode)
* [Index Mode](#index-mode)
@@ -46,7 +49,7 @@ Building changelog information in a form that is human digestible but still cont
All of these methods have their pros and cons. Some have issues with accuracy. Some have issues with lack of details. None of these methods seem to cover all of the needs of many projects and are full of potential pitfalls.
-In order to solve these problems, releasedocmaker was written to automatically generate a changelog and release notes by querying Apache's JIRA instance.
+In order to solve these problems, `releasedocmaker` was written to automatically generate a changelog and release notes by querying Apache's JIRA instance.
# Requirements
@@ -70,7 +73,7 @@ This is similar to the JIRA "Release Notes" button but is in tabular format and
* RELEASENOTES.md
-If your JIRA project supports the release note field, this will contain any JIRA mentioned in the CHANGELOG that is either an incompatible change or has a release note associated with it. If your JIRA project does not support the release notes field, this will be the description field.
+If your JIRA project supports the release note field, this will contain any JIRA mentioned in the `CHANGELOG` that is either an incompatible change or has a release note associated with it. If your JIRA project does not support the release notes field, this will be the description field.
For example, to build the release documentation for HBase v1.2.0:
@@ -87,7 +90,7 @@ remaining text
# Authentication
-releasedocmaker supports very simple Basic authentication. This is accomplished by adding two environment variables to your shell environment:
+`releasedocmaker` supports very simple Basic authentication. This is accomplished by adding two environment variables to your shell environment:
```bash
RDM_JIRA_USERNAME='jirausername'
@@ -108,21 +111,19 @@ Now instead of "HBASE", it will use "Apache HBase" for some titles and headers.
# Versioned Files and Directories
-It is sometimes useful to create the CHANGELOG and RELEASENOTES with versions attached. `releasedocmaker` supports both independently.
+It is sometimes useful to create the `CHANGELOG` and `RELEASENOTES` with versions attached. `releasedocmaker` supports both independently.
```bash
$ releasedocmaker --project HBASE --version 1.2.0 --fileversions
```
-This command line will now create CHANGELOG.1.2.0.md and RELEASENOTES.1.2.0.md files.
-
+This command line will now create `CHANGELOG.1.2.0.md` and `RELEASENOTES.1.2.0.md` files.
```bash
$ releasedocmaker --project HBASE --version 1.2.0 --dirversions
```
-This command line will now create a directory called 1.2.0 and inside will be the CHANGELOG.md and RELEASENOTES.md files.
-
+This command line will now create a directory called 1.2.0 and inside will be the `CHANGELOG.md` and `RELEASENOTES.md` files.
Using both at the same time...
@@ -130,8 +131,7 @@ Using both at the same time...
$ releasedocmaker --project HBASE --version 1.2.0 --fileversions --dirversions
```
-... results in 1.2.0/CHANGELOG.1.2.0.md and 1.2.0/RELEASENOTES.1.2.0.md files.
-
+... results in `1.2.0/CHANGELOG.1.2.0.md` and `1.2.0/RELEASENOTES.1.2.0.md` files.
# Multiple Versions
@@ -143,19 +143,19 @@ $ releasedocmaker --project HBASE --version 1.0.0 --version 1.2.0 --dirversions
This will create the files for versions 1.0.0 and versions 1.2.0 in their own directories.
-But what if the version numbers are not known? releasedocmaker can also generate version data based upon ranges:
+But what if the version numbers are not known? `releasedocmaker` can also generate version data based upon ranges:
```bash
$ releasedocmaker --project HBASE --version 1.0.0 --version 1.2.0 --range --fileversions
```
-In this form, releasedocmaker will query JIRA, discover all versions that alphabetically appear to be between 1.0.0 and 1.2.0, inclusive, and generate all of the relative release documents. This is especially useful when bootstrapping an existing project.
+In this form, `releasedocmaker` will query JIRA, discover all versions that alphabetically appear to be between 1.0.0 and 1.2.0, inclusive, and generate all of the relative release documents. This is especially useful when bootstrapping an existing project.
# Unreleased Dates
For released versions, releasedocmaker will pull the date of the release from JIRA. However, for unreleased versions it marks the release as "Unreleased". This can be inconvenient when actually building a release and wanting to include it inside the source package.
-The --usetoday option can be used to signify that instead of using Unreleased, releasedocmaker should use today's date.
+The `--usetoday` option can be used to signify that instead of using Unreleased, `releasedocmaker` should use today's date.
```bash
$ releasedocmaker --project HBASE --version 1.0.0 --usetoday
@@ -165,11 +165,11 @@ After using this option and release, don't forget to change JIRA's release date
# Sorted Output
-Different projects may find one type of sort better than another, depending upon their needs. releasedocmaker supports two types of sorts and each provides two different options in the direction for that sort.
+Different projects may find one type of sort better than another, depending upon their needs. `releasedocmaker` supports two types of sorts and each provides two different options in the direction for that sort.
## Resolution Date-base Sort
-By default, releasedocmaker will sort the output based upon the resolution date of the issue starting with older resolutions. This is the same as giving these options:
+By default, `releasedocmaker` will sort the output based upon the resolution date of the issue starting with older resolutions. This is the same as giving these options:
```bash
$ releasedocmaker --project falcon --version 0.6 --sorttype resolutiondate --sortorder older
@@ -203,15 +203,15 @@ In the case of multiple projects given on the command line, the projects will be
# Backward Incompatible Changes
-To check if an issue is backward-incompatible the releasedocmaker script first checks the "Hadoop Flags" field in the
+To check if an issue is backward-incompatible the `releasedocmaker` script first checks the "Hadoop Flags" field in the
issue. If this field is found to be blank then it searches for the 'backward-incompatible' label. You can override the
-default value for this label by using --incompatiblelabel option e.g.
+default value for this label by using `--incompatiblelabel` option e.g.
```bash
$ releasedocmaker --project falcon --version 0.6 --incompatiblelabel not-compatible
```
-or equivalently using the shorter -X option
+or equivalently using the shorter `-X` option
```bash
$ releasedocmaker --project falcon --version 0.6 -X not-compatible
@@ -219,9 +219,9 @@ $ releasedocmaker --project falcon --version 0.6 -X not-compatible
# Lint Mode
-In order to ensure proper formatting while using mvn site, releasedocmaker puts in periods (.) for fields that are empty or unassigned. This can be unsightly and not proper for any given project. There are also other things, such as missing release notes for incompatible changes, that are less than desirable.
+In order to ensure proper formatting while using mvn site, `releasedocmaker` puts in periods (.) for fields that are empty or unassigned. This can be unsightly and not proper for any given project. There are also other things, such as missing release notes for incompatible changes, that are less than desirable.
-In order to help release managers from having to scan through potentially large documents, releasedocmaker features a lint mode, triggered via --lint:
+In order to help release managers from having to scan through potentially large documents, `releasedocmaker` features a lint mode, triggered via `--lint`:
```bash
$ releasedocmaker --project HBASE --version 1.0.0 --lint
@@ -234,12 +234,12 @@ This will do the normal JIRA querying, looking for items it considers problemati
There is basic support for an autoindexer. It will create two files that contain links to all directories that have a major.minor*-style version numbering system.
For example directories with names like 0.6, 1.2.2, 1.2alpha etc. will all be linked.
- * index.md: a file suitable for conversion to HTML via mvn site
- * README.md: a file suitable for display on Github and other Markdown rendering websites
+* `index.md`: a file suitable for conversion to HTML via mvn site
+* `README.md`: a file suitable for display on Github and other Markdown rendering websites
# Release Version
-You can find the version of the releasedocmaker that you are using by giving the -V option. This may be helpful in finding documentation for the version you are using.
+You can find the version of the `releasedocmaker` that you are using by giving the `-V` option. This may be helpful in finding documentation for the version you are using.
```bash
$ releasedocmaker -V
diff --git a/asf-site-src/source/documentation/in-progress/shelldocs.md b/asf-site-src/source/documentation/in-progress/shelldocs.md
index 6627bc5..56db17b 100644
--- a/asf-site-src/source/documentation/in-progress/shelldocs.md
+++ b/asf-site-src/source/documentation/in-progress/shelldocs.md
@@ -17,9 +17,7 @@
under the License.
-->
-shelldocs
-===============
-
+# shelldocs
<!-- MarkdownTOC levels="1,2" autolink="true" -->
@@ -37,10 +35,9 @@ shelldocs
<!-- /MarkdownTOC -->
-
# Purpose
-Some projects have complex shell functions that act as APIs. `shelldocs` provides an annotation system similar to JavaDoc. It allows a developer to autogenerate MultiMarkdown documentation files as input to other processing systems.
+Some projects have complex shell functions that act as APIs. `shelldocs` provides an annotation system similar to JavaDoc. It allows a developer to auto-generate MultiMarkdown documentation files as input to other processing systems.
# Requirements
diff --git a/asf-site-src/source/documentation/in-progress/yetus-maven-plugin.md b/asf-site-src/source/documentation/in-progress/yetus-maven-plugin.md
index 8cfa6d9..1b77ade 100644
--- a/asf-site-src/source/documentation/in-progress/yetus-maven-plugin.md
+++ b/asf-site-src/source/documentation/in-progress/yetus-maven-plugin.md
@@ -17,8 +17,7 @@
under the License.
-->
-Yetus Maven Plug-in
-===================
+# Yetus Maven Plug-in
<!-- MarkdownTOC levels="1,2" autolink="true" -->
@@ -46,6 +45,7 @@ As part of building Apache Yetus, we needed some portable functionality that we
Apache Yetus builds wrappers in `bin/` that point to executables in `lib/`. This goal provides a way to do this generically, including providing the capability to put a license file in the wrapper.
+```xml
<plugin>
<groupId>org.apache.yetus</groupId>
<artifactId>yetus-maven-plugin</artifactId>
@@ -63,6 +63,7 @@ Apache Yetus builds wrappers in `bin/` that point to executables in `lib/`. Thi
</execution>
</executions>
</plugin>
+```
This example will take all the files located in `${project.build.directory}/dist/lib/shelldocs/` and create wrappers in `${project.build.directory}/dist/bin` with any extensions stripped off. If the `${project.build.directory}/dist/lib/shelldocs/` contains the file `shelldocs.py`, then the `bin/shelldocs` wrapper will look like this:
@@ -94,6 +95,7 @@ If no license is wanted, then set `license` to the string `none`.
Since Java 7, there has been a portable way to build symlinks. Unfortunately, standard plug-ins like the `maven-antrun-plugin` have not been updated to include the symlink task. The `yetus-maven-plugin` now exposes this functionality via the `symlink` goal:
+```xml
<plugin>
<groupId>org.apache.yetus</groupId>
<artifactId>yetus-maven-plugin</artifactId>
@@ -112,6 +114,7 @@ Since Java 7, there has been a portable way to build symlinks. Unfortunately, s
</configuration>
</execution>
</plugin>
+```
Available configuration options:
@@ -127,6 +130,7 @@ Available configuration options:
Maven surefire (as of at least 2.x and earlier versions) has calculations to determine the number of tests to run in parallel. However, the result is not shared in a way that allows creating directory structures before execution. For specific build flows, this is problematic.
+```xml
<plugin>
<groupId>org.apache.yetus</groupId>
<artifactId>yetus-maven-plugin</artifactId>
@@ -142,6 +146,7 @@ Maven surefire (as of at least 2.x and earlier versions) has calculations to det
</configuration>
</execution>
</plugin>
+```
Available configuration options:
@@ -157,6 +162,7 @@ By default, `forkCount` is inherited from surefire and therefore follows the sam
This goal runs releasedocmaker without the need to download or build an Apache Yetus tarball. Instead, yetus-maven-plugin contains all the necessary components in a native maven way!
+```xml
<plugin>
<groupId>org.apache.yetus</groupId>
<artifactId>yetus-maven-plugin</artifactId>
@@ -177,6 +183,7 @@ This goal runs releasedocmaker without the need to download or build an Apache Y
</configuration>
</execution>
</plugin>
+```
The configuration options generally map 1:1 to the `releasedocmaker` executable's options. Unless otherwise specified, defaults are set by the actual executable.
@@ -204,6 +211,7 @@ The configuration options generally map 1:1 to the `releasedocmaker` executable'
Similar to the `releasedocmaker` goal, the `shelldocs` goal runs the Apache Yetus `shelldocs` utility against a given set of input files or directories and generates a single output MultiMarkdown file:
+```xml
<plugin>
<groupId>org.apache.yetus</groupId>
<artifactId>yetus-maven-plugin</artifactId>
@@ -216,8 +224,7 @@ Similar to the `releasedocmaker` goal, the `shelldocs` goal runs the Apache Yetu
</goals>
</execution>
</plugin>
-
-
+```
The configuration options generally map 1:1 to the `shelldocs` executable's options. Unless otherwise specified, defaults are set by the actual executable.
diff --git a/asf-site-src/source/index.html.md b/asf-site-src/source/index.html.md
index bd7ec5b..f5e7af6 100644
--- a/asf-site-src/source/index.html.md
+++ b/asf-site-src/source/index.html.md
@@ -17,6 +17,7 @@
under the License.
-->
+<!-- markdownlint-disable no-inline-html first-line-heading -->
<div class="masthead">
<div class="jumbotron">
<h1>Apache Yetus</h1>
diff --git a/asf-site-src/source/mailinglists.html.md b/asf-site-src/source/mailinglists.html.md
index 40f513a..43370a8 100644
--- a/asf-site-src/source/mailinglists.html.md
+++ b/asf-site-src/source/mailinglists.html.md
@@ -18,6 +18,8 @@
-->
# Community
+
+<!-- markdownlint-disable no-inline-html -->
<div class="row-fluid">
<div class="col-lg-6">
<h3>Mailing Lists</h3>
diff --git a/asf-site-src/source/yetus-docker-image.md b/asf-site-src/source/yetus-docker-image.md
index 963cbf5..822b9f0 100644
--- a/asf-site-src/source/yetus-docker-image.md
+++ b/asf-site-src/source/yetus-docker-image.md
@@ -17,25 +17,30 @@
under the License.
-->
-Convenience Docker Hub Images
-===========================
+# Convenience Docker Hub Images
+
+<!-- MarkdownTOC levels="1,2" autolink="true" -->
+
+* [File Access](#file-access)
+* [A Note About Precommit](#a-note-about-precommit)
+
+<!-- /MarkdownTOC -->
While not official releases, the Apache Yetus project makes available two types of images on hub.docker.com:
- * apache/yetus-base
+* apache/yetus-base
- This image is the same as the 'built-in' Dockerfile when `--docker` is used without `--dockerfile` on the precommit command line. It includes all of the pre-requisites as needed by the various Apache Yetus components. It is located in `precommit/src/main/shell/test-patch-docker/`.
+ This image is the same as the 'built-in' Dockerfile when `--docker` is used without `--dockerfile` on the precommit command line. It includes all of the pre-requisites as needed by the various Apache Yetus components. It is located in `precommit/src/main/shell/test-patch-docker/`.
- * apache/yetus
+* apache/yetus
- This image is the same as apache/yetus-base but includes a pre-built version of Apache Yetus as part of the base OS image. In other words, qbt, releasedocmaker, shelldocs, test-patch, etc., are in /usr/bin and available in the default path. It is generated from the Dockerfile located in the root of the source and is built with the options provided in the hooks directory.
+ This image is the same as apache/yetus-base but includes a pre-built version of Apache Yetus as part of the base OS image. In other words, qbt, releasedocmaker, shelldocs, test-patch, etc., are in /usr/bin and available in the default path. It is generated from the Dockerfile located in the root of the source and is built with the options provided in the hooks directory.
Both images should be suitable to be used as a building block or even directly if your build environment needs no other dependencies. These images are especially useful for various CI systems that require a Docker image to be used.
Images are tagged such that 'master' represents the last successful Docker image build of the master branch. Images based off of the official source releases are tagged with a matching version number (e.g., 0.9.0). There is no 'latest' tagged image. It is recommended that users choose a stable tag so as not to be surprised by incompatible changes.
-File Access
-========
+# File Access
All of the executables that Apache Yetus provides requires access to one or more directories. These directories should be provided on the command line via the volume flags to docker run. For example, to run shelldocs against test-patch.sh:
@@ -64,6 +69,7 @@ docker run \
--patch-dir=/patchdir \
--project=yetus
```
+
If your project needs additional dependencies, it is trivial to build off of the Apache Yetus image:
```Dockerfile
@@ -74,13 +80,13 @@ RUN apt-get -q update && apt-get -q install -y \
valgrind \
zlib1g-dev
```
- ```bash
+
+```bash
docker build -t project/build:0.9.0 -f .
```
This example builds a docker image based off of Apache Yetus 0.9.0 but with the additions of clang, some development libraries, and valgrind. Now project/build:0.9.0 can be used instead of apache/yetus:0.9.0 since it has all of Apache Yetus and the additions needed by our project.
-A Note About Precommit
-================
+# A Note About Precommit
test-patch and friends have direct support for Docker outside of the convenience images. That information is covered in-depth in the[precommit-docker](../precommit-docker) section.
diff --git a/precommit/src/main/shell/coprocs.d/README.md b/precommit/src/main/shell/coprocs.d/README
similarity index 97%
rename from precommit/src/main/shell/coprocs.d/README.md
rename to precommit/src/main/shell/coprocs.d/README
index e960cad..ef64d67 100755
--- a/precommit/src/main/shell/coprocs.d/README.md
+++ b/precommit/src/main/shell/coprocs.d/README
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -22,4 +21,4 @@
# bash v3 and lower will treat coproc commands as syntax errors
# therefore, ALL functions which activate coprocs are located
-# here
\ No newline at end of file
+# here
diff --git a/precommit/src/main/shell/test-patch-docker/Dockerfile b/precommit/src/main/shell/test-patch-docker/Dockerfile
index c563dfd..52ba190 100644
--- a/precommit/src/main/shell/test-patch-docker/Dockerfile
+++ b/precommit/src/main/shell/test-patch-docker/Dockerfile
@@ -240,12 +240,12 @@ RUN curl -L -s -S \
###
# Install npm and JSHint
###
-# hadolint ignore=DL3008,DL3016
+# hadolint ignore=DL3008
RUN curl -sL https://deb.nodesource.com/setup_11.x | bash - \
&& apt-get -q install --no-install-recommends -y nodejs \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* && \
- npm install -g jshint
+ npm install -g jshint@2.10.2 markdownlint-cli@0.15.0
####
# YETUS CUT HERE
diff --git a/precommit/src/main/shell/test-patch.d/markdownlint.sh b/precommit/src/main/shell/test-patch.d/markdownlint.sh
new file mode 100755
index 0000000..884836d
--- /dev/null
+++ b/precommit/src/main/shell/test-patch.d/markdownlint.sh
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# no public APIs here
+# SHELLDOC-IGNORE
+
+add_test_type markdownlint
+
+MARKDOWNLINT_TIMER=0
+MARKDOWNLINT=${MARKDOWNLINT:-$(command -v markdownlint 2>/dev/null)}
+
+# files that are going to get markdownlint'd
+MARKDOWNLINT_CHECKFILES=()
+
+function markdownlint_filefilter
+{
+ declare filename=$1
+
+ if [[ ${filename} =~ \.md$ ]]; then
+ add_test markdownlint
+ yetus_add_array_element MARKDOWNLINT_CHECKFILES "${filename}"
+ fi
+}
+
+function markdownlint_precheck
+{
+ if ! verify_command "markdownlint" "${MARKDOWNLINT}"; then
+ add_vote_table 0 markdownlint "markdownlint was not available."
+ delete_test markdownlint
+ fi
+}
+
+function markdownlint_logic
+{
+ declare repostatus=$1
+ declare i
+ declare fn
+ declare output
+
+ pushd "${BASEDIR}" >/dev/null || return 1
+
+ for i in "${MARKDOWNLINT_CHECKFILES[@]}"; do
+ if [[ -f "${i}" ]]; then
+ fn=""
+ while read -r; do
+ if [[ -z "${fn}" ]]; then
+ fn=$REPLY
+ elif [[ -n "${REPLY}" ]]; then
+ # (space)line:col(space)error/warning(space)text
+ output=$(echo "${REPLY}" | awk '{$1=$1":"; $2=$2":"; print $0;}')
+ # fn:line:col:(space)error/warning:(space)text
+ echo "${fn}:${output}" >> "${PATCH_DIR}/${repostatus}-markdownlint-result.txt"
+ fi
+ done < <("${MARKDOWNLINT}" "${i}" 2>&1)
+ fi
+ done
+ popd > /dev/null || return 1
+}
+
+function markdownlint_preapply
+{
+ if ! verify_needed_test markdownlint; then
+ return 0
+ fi
+
+ big_console_header "markdownlint plugin: ${PATCH_BRANCH}"
+
+ start_clock
+
+ markdownlint_logic branch
+
+ # keep track of how much as elapsed for us already
+ MARKDOWNLINT_TIMER=$(stop_clock)
+ return 0
+}
+
+function markdownlint_calcdiffs
+{
+ column_calcdiffs "$@"
+}
+
+function markdownlint_postapply
+{
+ declare i
+ declare numPrepatch
+ declare numPostpatch
+ declare diffPostpatch
+ declare fixedpatch
+ declare statstring
+ declare version
+
+ if ! verify_needed_test markdownlint; then
+ return 0
+ fi
+
+ big_console_header "markdownlint plugin: ${BUILDMODE}"
+
+ start_clock
+
+ # add our previous elapsed to our new timer
+ # by setting the clock back
+ offset_clock "${MARKDOWNLINT_TIMER}"
+
+ markdownlint_logic patch
+
+ # shellcheck disable=SC2016
+ version=$("${MARKDOWNLINT}" --version 2>&1)
+ add_version_data markdownlint "${version#* }"
+
+ calcdiffs \
+ "${PATCH_DIR}/branch-markdownlint-result.txt" \
+ "${PATCH_DIR}/patch-markdownlint-result.txt" \
+ markdownlint \
+ > "${PATCH_DIR}/diff-patch-markdownlint.txt"
+
+ # shellcheck disable=SC2016
+ numPrepatch=$(wc -l "${PATCH_DIR}/branch-markdownlint-result.txt" | "${AWK}" '{print $1}')
+
+ # shellcheck disable=SC2016
+ numPostpatch=$(wc -l "${PATCH_DIR}/patch-markdownlint-result.txt" | "${AWK}" '{print $1}')
+
+ # shellcheck disable=SC2016
+ diffPostpatch=$(wc -l "${PATCH_DIR}/diff-patch-markdownlint.txt" | "${AWK}" '{print $1}')
+
+
+ ((fixedpatch=numPrepatch-numPostpatch+diffPostpatch))
+
+ statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" )
+
+ if [[ ${diffPostpatch} -gt 0 ]] ; then
+ add_vote_table -1 markdownlint "${BUILDMODEMSG} ${statstring}"
+ add_footer_table markdownlint "@@BASE@@/diff-patch-markdownlint.txt"
+ bugsystem_linecomments_queue "markdownlint" "${PATCH_DIR}/diff-patch-markdownlint.txt"
+ return 1
+ elif [[ ${fixedpatch} -gt 0 ]]; then
+ add_vote_table +1 markdownlint "${BUILDMODEMSG} ${statstring}"
+ return 0
+ fi
+
+ add_vote_table +1 markdownlint "There were no new markdownlint issues."
+ return 0
+}
+
+function markdownlint_postcompile
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ markdownlint_preapply
+ else
+ markdownlint_postapply
+ fi
+}