You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by aw...@apache.org on 2015/09/09 19:21:48 UTC
[2/2] hadoop git commit: HADOOP-12257. rework build tool support;
add gradle; add scala (aw)
HADOOP-12257. rework build tool support; add gradle; add scala (aw)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1e2eeb07
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1e2eeb07
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1e2eeb07
Branch: refs/heads/HADOOP-12111
Commit: 1e2eeb07c0a6e56bb6764a76195a9836272b660e
Parents: 5e93128
Author: Allen Wittenauer <aw...@apache.org>
Authored: Wed Sep 9 10:21:32 2015 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Wed Sep 9 10:21:32 2015 -0700
----------------------------------------------------------------------
dev-support/docs/precommit-advanced.md | 99 +-
dev-support/docs/precommit-architecture.md | 42 +-
dev-support/docs/precommit-basic.md | 5 +-
dev-support/docs/precommit-bugsystems.md | 46 +
dev-support/docs/precommit-buildtools.md | 100 +
dev-support/personality/bigtop.sh | 67 +
dev-support/personality/flink.sh | 61 +-
dev-support/personality/hadoop.sh | 27 +-
dev-support/personality/hbase.sh | 113 +-
dev-support/personality/kafka.sh | 58 +
dev-support/personality/pig.sh | 2 +-
dev-support/personality/samza.sh | 26 +
dev-support/personality/tajo.sh | 40 +-
dev-support/personality/tez.sh | 44 -
dev-support/test-patch.d/ant.sh | 183 ++
dev-support/test-patch.d/asflicense.sh | 25 +-
dev-support/test-patch.d/author.sh | 52 +
dev-support/test-patch.d/builtin-bugsystem.sh | 0
dev-support/test-patch.d/builtin-personality.sh | 140 +-
dev-support/test-patch.d/checkstyle.sh | 59 +-
dev-support/test-patch.d/findbugs.sh | 53 +-
dev-support/test-patch.d/gradle.sh | 248 +++
dev-support/test-patch.d/java.sh | 158 ++
dev-support/test-patch.d/jira.sh | 2 +-
dev-support/test-patch.d/maven.sh | 328 +++
dev-support/test-patch.d/perlcritic.sh | 11 +
dev-support/test-patch.d/pylint.sh | 11 +
dev-support/test-patch.d/rubocop.sh | 11 +
dev-support/test-patch.d/ruby-lint.sh | 11 +
dev-support/test-patch.d/scala.sh | 77 +
dev-support/test-patch.d/shellcheck.sh | 11 +
dev-support/test-patch.d/test4tests.sh | 58 +
dev-support/test-patch.d/whitespace.sh | 20 +-
dev-support/test-patch.d/xml.sh | 15 +-
dev-support/test-patch.sh | 1934 +++++++-----------
35 files changed, 2455 insertions(+), 1682 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/docs/precommit-advanced.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-advanced.md b/dev-support/docs/precommit-advanced.md
index 7830afe..83bda30 100644
--- a/dev-support/docs/precommit-advanced.md
+++ b/dev-support/docs/precommit-advanced.md
@@ -16,8 +16,6 @@ test-patch
==========
* [Docker Support](#Docker_Support)
-* [Maven Specific](#Maven_Specific)
-* [Ant Specific](#Ant_Specific)
* [Plug-ins](#Plug-ins)
* [Configuring for Other Projects](#Configuring_for_Other_Projects)
* [Important Variables](#Important_Variables)
@@ -32,107 +30,71 @@ NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX)
Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --jenkins mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system.
-# Maven Specific
-
-## Command Arguments
-
-test-patch always passes --batch-mode to maven to force it into non-interactive mode. Additionally, some tests will also force -fae in order to get all of messages/errors during that mode. It *does not* pass -DskipTests. Additional arguments should be handled via the personality.
-
-## Test Profile
-
-By default, test-patch will pass -Ptest-patch to Maven. This will allow you to configure special actions that should only happen when running underneath test-patch.
-
-# Ant Specific
-
-## Command Arguments
-
-test-patch always passes -noinput to Ant. This force ant to be non-interactive.
# Plug-ins
-test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary.
+test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. Different plug-ins have specific functions for that particular functionality. In this document, the common functions as well as test functions are covered. See other documentat for pertinent information for the other plug-in types.
-## Test Plug-ins
+## Common Plug-in Functions
-Every test plugin must have one line in order to be recognized:
+Every plug-in must have one line in order to be recognized, usually an 'add' statement. Test plug-ins, for example, have this statement:
```bash
add_plugin <pluginname>
```
-This function call registers the `pluginname` so that test-patch knows that it exists. This plug-in name also acts as the key to the custom functions that you can define. For example:
+This function call registers the `pluginname` so that test-patch knows that it exists. The `pluginname` also acts as the key to the custom functions that you can define. For example:
```bash
function pluginname_filefilter
```
-This function gets called for every file that a patch may contain. This allows the plug-in author to determine if this plug-in should be called, what files it might need to analyze, etc.
+defines the filefilter for the `pluginname` plug-in.
Similarly, there are other functions that may be defined during the test-patch run:
-* pluginname\_postcheckout
- - executed prior to the patch being applied but after the git repository is setup. This is useful for any early error checking that might need to be done before any heavier work.
-
-* pluginname\_preapply
- - executed prior to the patch being applied. This is useful for any "before"-type data collection for later comparisons.
-
-* pluginname\_postapply
- - executed after the patch has been applied. This is useful for any "after"-type data collection.
-
-* pluginname\_postinstall
- - executed after the mvn install test has been done. If any tests require the Maven repository to be up-to-date with the contents of the patch, this is the place.
-
-* pluginname\_tests
- - executed after the unit tests have completed.
-
-If the plug-in has some specific options, one can use following functions:
-
-* pluginname\_usage
-
- - executed when the help message is displayed. This is used to display the plug-in specific options for the user.
-
-* pluginname\_parse\_args
-
- - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
-
HINT: It is recommended to make the pluginname relatively small, 10 characters at the most. Otherwise, the ASCII output table may be skewed.
-## Bug System Plug-ins
-
-Similar to tests, the ability to add support for bug tracking systems is also handled via a plug-in mechanism.
-
-* pluginname_usage
+* pluginname\_initialize
+ - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
+* pluginname\_usage
- executed when the help message is displayed. This is used to display the plug-in specific options for the user.
* pluginname\_parse\_args
+ - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
- - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
-
-
-* pluginname\_locate\_patch
+* pluginname\_precheck
+ - executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch.
- - Given input from the user, download the patch if possible.
+* pluginname\_precompile
+ - executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process.
-* pluginname\_determine\_branch
+* pluginname\_compile
+ - executed immediately after the actual compilation. This is step is intended to be used to verify the results and add extra checking of the compile phase and it's stdout/stderr.
- - Using any heuristics available, return the branch to process, if possible.
+* pluginname\_postcompile
+ - This step happens after the compile phase.
-* pluginname\_determine\_issue
+* pluginname\_rebuild
+ - Any non-unit tests that require the source to be rebuilt in a destructive way should be run here.
- - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table.
+Test Plug-ins
+=============
-* pluginname\_writecomment
+Plugins geared towards independent tests are registed via:
- - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately.
-* pluginname\_linecomments
+```bash
+add_plugin <pluginname>
+```
- - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
++ pluginname\_filefilter
+ - executed while determine which files trigger which tests. This function should use 'add_test pluginname' to add the plug-in to the test list.
-* pluginname\_finalreport
+* pluginname\_tests
+ - executed after the unit tests have completed.
- - Write the final result table to the bug system.
# Configuring for Other Projects
@@ -197,7 +159,6 @@ The second is `personality_enqueue_module`. This function takes two parameters.
For example, let's say your project uses a special configuration to skip unit tests (-DskipTests). Running unit tests during a javadoc build isn't very useful and wastes a lot of time. We can write a simple personality check to disable the unit tests:
-
```bash
function personality_modules
{
@@ -214,8 +175,6 @@ function personality_modules
This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool.
-
-
# Important Variables
There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/docs/precommit-architecture.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-architecture.md b/dev-support/docs/precommit-architecture.md
index cd527ae..e0a178a 100644
--- a/dev-support/docs/precommit-architecture.md
+++ b/dev-support/docs/precommit-architecture.md
@@ -30,49 +30,55 @@
test-patch works effectively under several different phases:
-## Setup
+## Initialize
This is where test-patch configures and validates the environment. Some things done in this phase:
* Defaults
* Parameter handling
-* Importing plugins and personalities
+* Importing plug-ins and personalities
* Docker container launching
* Re-exec support
* Patch file downloading
* git repository management (fresh pull, branch switching, etc)
-## Post-checkout
+## Precheck
Checks done here are *fatal*.
This acts as a verification of all of the setup parts and is the final place to short-cut the full test cycle. The most significant built-in check done here is verifying the patch file is a valid.
-## Pre-apply
+## Patch File Tests
-This is where the 'before' work is handled. Some things that typically get checked in this phase:
+Tests that only require the patch file are run. Note that the repository is still from the initial checkout!
+
+## Compile Cycle (Branch)
+
+When compilation must be done, we follow these five steps:
+
+* The list of modules that require analysis is built.
+* A precompile step to set things up for the actual compile
+* The actual compile
+* A postcompile to do analysis on the output of that compile phase
+* A rebuild phase to run tests that require recompiles
+
+The first time this is done is with the pristine checkout. This is called the "branch compile". For this pass, this is where the 'before' work is handled. Some things that typically get checked in this phase:
* The first pass of files and modules that will get patched
* Validation and information gathering of the source tree pre-patch
-* Author checks
-* Check for modified unit tests
-
-## Patch is Applied
+* javadoc, scaladoc, etc
-The patch gets applied. Then a second pass to determine which modules and files have been changed in order to handle any modules that might have added or moved.
+## Distribution Clean
-## Post-apply
+This step is to wipe the repository clean back to a pristine state such that the previous cycle will not impact the next cycle.
-Now that the patch has been applied, many of the same checks performed in the Pre-apply step are done again to build an 'after' picture.
+## Patch Application
-## Post-install
+The patch gets applied.
-Some tests only work correctly when the repo is up-to-date. So
-mvn install is run to update the local repo and we enter this phase. Some example tests performed here:
+## Compile Cycle (Patch)
-* javadoc
-* Findbugs
-* Maven eclipse integration still works
+Now that the patch has been applied the steps to compile we outlined in the compilation (branch) phase are repeated but with the patch applied. This is where a lot of 'after' checks are performed.
## Unit Tests
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/docs/precommit-basic.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-basic.md b/dev-support/docs/precommit-basic.md
index a612214..9cb4035 100644
--- a/dev-support/docs/precommit-basic.md
+++ b/dev-support/docs/precommit-basic.md
@@ -37,7 +37,7 @@ This is a modification to Hadoop's version of test-patch so that we may bring to
test-patch has the following requirements:
-* Ant- or Maven-based project (and ant/maven installed)
+* A project with a supported build tool (ant, gradle, maven, ...)
* git-based project (and git 1.7.3 or higher installed)
* bash v3.2 or higher
* findbugs 3.x installed
@@ -52,7 +52,7 @@ test-patch has the following requirements:
* file command
* smart-apply-patch.sh (included!)
-Maven plugins requirements:
+Maven plug-ins requirements:
* Apache RAT
* Apache FindBugs
@@ -61,6 +61,7 @@ Optional:
* JIRA-based issue tracking
* GitHub-based issue tracking
+* Some other supported bug system
The locations of these files are (mostly) assumed to be in the file path, but may be overridden via command line options. For Solaris and Solaris-like operating systems, the default location for the POSIX binaries is in /usr/xpg4/bin and the default location for the GNU binaries is /usr/gnu/bin.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/docs/precommit-bugsystems.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-bugsystems.md b/dev-support/docs/precommit-bugsystems.md
new file mode 100644
index 0000000..6af1266
--- /dev/null
+++ b/dev-support/docs/precommit-bugsystems.md
@@ -0,0 +1,46 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Bug System Support
+==================
+
+test-patch has the ability to support multiple bug systems. Bug tools have some extra hooks to fetch patches, line-level reporting, and posting a final report. Every bug system plug-in must have one line in order to be recognized:
+
+```bash
+add_bugsystem <pluginname>
+```
+
+* pluginname\_locate\_patch
+
+ - Given input from the user, download the patch if possible.
+
+* pluginname\_determine\_branch
+
+ - Using any heuristics available, return the branch to process, if possible.
+
+* pluginname\_determine\_issue
+
+ - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table.
+
+* pluginname\_writecomment
+
+ - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately.
+
+* pluginname\_linecomments
+
+ - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
+
+* pluginname\_finalreport
+
+ - Write the final result table to the bug system.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/docs/precommit-buildtools.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-buildtools.md b/dev-support/docs/precommit-buildtools.md
new file mode 100644
index 0000000..d14f34b
--- /dev/null
+++ b/dev-support/docs/precommit-buildtools.md
@@ -0,0 +1,100 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Build Tool Support
+===================
+
+test-patch has the ability to support multiple build tools. Build tool plug-ins have some extra hooks to do source and object maintenance at key points. Every build tool plug-in must have one line in order to be recognized:
+
+```bash
+add_build_tool <pluginname>
+```
+
+# Global Variables
+
+* BUILDTOOLCWD
+
+ - If the build tool does not always run from the ${BASEDIR} and instead needs to change the current working directory to work on a specific module, then set this to true. The default is false.
+
+* UNSUPPORTED_TEST
+
+ - If pluginname\_modules\_worker is given a test type that is not supported by the build system, set UNSUPPORTED_TEST=true. If it is supported, set UNSUPPORTED_TEST=false.
+
+For example, the gradle build tool does not have a standard way to execute checkstyle. So when checkstyle is requested, gradle\_modules\_worker sets UNSUPPORTED_TEST to true and returns out of the routine.
+
+# Required Functions
+
+* pluginname\_buildfile
+
+ - This should be an echo of the file that controls the build system. This is used for module determination.
+
+* pluginname\_executor
+
+ - This should be an echo of how to run the build tool, any extra arguments, etc.
+
+* pluginname\_modules\_worker
+
+ - Input is the branch and the test being run. This should call modules_workers with the generic parts to run that test on the build system. For example, if it is convention to use 'test' to trigger 'unit' tests, then module_workers should be called with 'test' appended onto its normal parameters.
+
+* pluginname\_builtin_personality\_modules
+
+ - Default method to determine how to enqueue modules for processing. Note that personalities may override this function.
+
+* pluginname\_builtin_personality\_file\_tests
+
+ - Default method to determine which tests to trigger. Note that personalities may override this function.
+
+# Optional Functions
+
+* pluginname\_postapply\_install
+
+ - After the install step, this allows the build tool plug-in to do extra work.
+
+* pluginname\_parse\_args
+
+ - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+
+* pluginname\_initialize
+
+ - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
+
+* pluginname\_count\_(test)\_probs
+
+ - Certain language test plug-ins require assistance from the build tool to count problems in the compile log due to some tools having custom handling for those languages. The test plug-in name should be in the (test) part of the function name.
+
+# Ant Specific
+
+## Command Arguments
+
+test-patch always passes -noinput to Ant. This forces ant to be non-interactive.
+
+# Gradle Specific
+
+The gradle plug-in always rebuilds the gradlew file and uses gradlew as the method to execute commands.
+
+# Maven Specific
+
+## Command Arguments
+
+test-patch always passes --batch-mode to maven to force it into non-interactive mode. Additionally, some tests will also force -fae in order to get all of messages/errors during that mode. Some tests are executed with -DskipTests. Additional arguments should be handled via the personality.
+
+## Test Profile
+
+By default, test-patch will pass -Ptest-patch to Maven. This will allow you to configure special actions that should only happen when running underneath test-patch.
+
+## Custom Maven Tests
+
+* Maven will trigger a maven install as part of the precompile.
+* Maven will test eclipse integration as part of the postcompile.
+* If src/site is modified, maven site tests are executed.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/bigtop.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/bigtop.sh b/dev-support/personality/bigtop.sh
new file mode 100755
index 0000000..e467bd1
--- /dev/null
+++ b/dev-support/personality/bigtop.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# shellcheck disable=SC2034
+PATCH_BRANCH_DEFAULT=master
+# shellcheck disable=SC2034
+JIRA_ISSUE_RE='^BIGTOP-[0-9]+$'
+# shellcheck disable=SC2034
+HOW_TO_CONTRIBUTE=""
+# shellcheck disable=SC2034
+BUILDTOOL=gradle
+# shellcheck disable=SC2034
+GITHUB_REPO="apache/bigtop"
+# shellcheck disable=SC2034
+BIGTOP_PUPPETSETUP=false
+
+add_plugin bigtop
+
+function bigtop_usage
+{
+ echo "Bigtop specific:"
+ echo "--bigtop-puppetsetup=[false|true] execute the bigtop dev setup (needs sudo to root)"
+}
+
+function bigtop_parse_args
+{
+ local i
+
+ for i in "$@"; do
+ case ${i} in
+ --bigtop-puppet=*)
+ BIGTOP_PUPPETSETUP=${i#*=}
+ ;;
+ esac
+ done
+}
+
+function bigtop_precheck_postinstall
+{
+ if [[ ${BIGTOP_PUPPETSETUP} = "true" ]]; then
+ pushd "${BASEDIR}" >/dev/null
+ echo_and_redirect "${PATCH_DIR}/bigtop-branch-toolchain.txt" "${GRADLEW}" toolchain
+ popd >/dev/null
+ fi
+}
+
+function bigtop_postapply_postinstall
+{
+ if [[ ${BIGTOP_PUPPETSETUP} = "true" ]]; then
+ pushd "${BASEDIR}" >/dev/null
+ echo_and_redirect "${PATCH_DIR}/bigtop-patch-toolchain.txt" "${GRADLEW}" toolchain
+ popd >/dev/null
+ fi
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/flink.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/flink.sh b/dev-support/personality/flink.sh
index 4b6c390..fbf0ba0 100755
--- a/dev-support/personality/flink.sh
+++ b/dev-support/personality/flink.sh
@@ -57,7 +57,7 @@ function flinklib_preapply
pushd "${BASEDIR}" >/dev/null
echo_and_redirect "${PATCH_DIR}/branch-flinklib-root.txt" \
- "${MVN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
+ "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
if [[ $? != 0 ]]; then
add_vote_table -1 flinklib "Unable to determine flink libs in ${PATCH_BRANCH}."
fi
@@ -78,7 +78,7 @@ function flinklib_postapply
pushd "${BASEDIR}" >/dev/null
echo_and_redirect "${PATCH_DIR}/patch-flinklib-root.txt" \
- "${MVN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
+ "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch
FLINK_POST_LIB_FILES=$(flinklib_count)
popd >/dev/null
@@ -97,57 +97,14 @@ function flinklib_postapply
return 0
}
-function personality_modules
+function flinklib_rebuild
{
- local repostatus=$1
- local testtype=$2
- local extra=""
-
- yetus_debug "Personality: ${repostatus} ${testtype}"
-
- clear_personality_queue
-
- case ${testtype} in
- mvninstall)
- if [[ ${repostatus} == branch ]]; then
- personality_enqueue_module . -DskipTests
- return
- fi
- return
- ;;
- asflicense)
- # this is very fast and provides the full path if we do it from
- # the root of the source
- personality_enqueue_module .
- return
- ;;
- unit)
- if [[ ${TEST_PARALLEL} == "true" ]] ; then
- extra="-Pparallel-tests"
- if [[ -n ${TEST_THREADS:-} ]]; then
- extra="${extra} -DtestsThreadCount=${TEST_THREADS}"
- fi
- fi
- ;;
- *)
- extra="-DskipTests"
- ;;
- esac
-
- for module in ${CHANGED_MODULES}; do
- # shellcheck disable=SC2086
- personality_enqueue_module ${module} ${extra}
- done
-}
-
-function personality_file_tests
-{
- local filename=$1
+ declare repostatus=$1
- yetus_debug "Using personality_file_tests, but calling the built-in:"
- builtin_personality_file_tests "${1}"
-
- if [[ ${filename} =~ \.scala$ ]]; then
- add_test unit
+ if [[ "${repostatus}" = branch ]]; then
+ flinklib_preapply
+ else
+ flinklib_postinstall
fi
}
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/hadoop.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/hadoop.sh b/dev-support/personality/hadoop.sh
index b3eb04a..0dd89db 100755
--- a/dev-support/personality/hadoop.sh
+++ b/dev-support/personality/hadoop.sh
@@ -34,6 +34,8 @@ function hadoop_module_manipulation
local startingmodules=${1:-normal}
local module
local hdfs_modules
+ local mapred_modules
+ local yarn_modules
local ordered_modules
local tools_modules
local passed_modules
@@ -67,9 +69,12 @@ function hadoop_module_manipulation
yetus_debug "hmm pre-ordering: ${startingmodules}"
- # yarn will almost always be after common in the sort order
- # so really just need to make sure that common comes before
- # everything else and tools comes last
+ # Hadoop's compilation rules are complex
+ # Precedence: common > [hdfs|yarn] > mapred > tools
+ # - everything depends upon common
+ # - hdfs needs common's native bits for unit tests
+ # - mapred depends upon yarn since it is a yarn app
+ # - tools can require anything
for module in ${passed_modules}; do
yetus_debug "Personality ordering ${module}"
@@ -85,12 +90,16 @@ function hadoop_module_manipulation
ordered_modules="${ordered_modules} ${module}"
elif [[ ${module} = hadoop-tools* ]]; then
tools_modules="${tools_modules} ${module}"
+ elif [[ ${module} = hadoop-mapreduce-project* ]]; then
+ mapred_modules="${mapred_modules} ${module}"
+ elif [[ ${module} = hadoop-yarn-project* ]]; then
+ yarn_modules="${yarn_modules} ${module}"
else
ordered_modules="${ordered_modules} ${module}"
fi
done
- HADOOP_MODULES="${ordered_modules} ${hdfs_modules} ${tools_modules}"
+ HADOOP_MODULES="${ordered_modules} ${hdfs_modules} ${yarn_modules} ${mapred_modules} ${tools_modules}"
yetus_debug "hmm out: ${HADOOP_MODULES}"
}
@@ -121,7 +130,7 @@ function hadoop_unittest_prereqs
pushd "${BASEDIR}/${module}" >/dev/null
# shellcheck disable=SC2086
echo_and_redirect "${PATCH_DIR}/maven-unit-prereq-${fn}-install.txt" \
- "${MVN}" "${MAVEN_ARGS[@]}" install -DskipTests ${flags}
+ "${MAVEN}" "${MAVEN_ARGS[@]}" install -DskipTests ${flags}
popd >/dev/null
fi
}
@@ -212,7 +221,7 @@ function personality_modules
ordering="union"
extra="-DskipTests"
;;
- javac)
+ compile)
ordering="union"
extra="-DskipTests"
needflags=true
@@ -223,6 +232,10 @@ function personality_modules
ordering="."
fi
;;
+ distclean)
+ ordering="."
+ extra="-DskipTests"
+ ;;
javadoc)
if [[ ${repostatus} = patch ]]; then
echo "javadoc pre-reqs:"
@@ -232,7 +245,7 @@ function personality_modules
pushd "${BASEDIR}/${i}" >/dev/null
echo "cd ${i}"
echo_and_redirect "${PATCH_DIR}/maven-${fn}-install.txt" \
- "${MVN}" "${MAVEN_ARGS[@]}" install
+ "${MAVEN}" "${MAVEN_ARGS[@]}" install
popd >/dev/null
done
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/hbase.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/hbase.sh b/dev-support/personality/hbase.sh
index 4f23679..120efa7 100755
--- a/dev-support/personality/hbase.sh
+++ b/dev-support/personality/hbase.sh
@@ -24,7 +24,7 @@ GITHUB_REPO="apache/hbase"
HOW_TO_CONTRIBUTE=""
# All supported Hadoop versions that we want to test the compilation with
-HADOOP2_VERSIONS="2.4.1 2.5.2 2.6.0"
+HBASE_HADOOP_VERSIONS="2.4.1 2.5.2 2.6.0"
# Override the maven options
MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}"
@@ -39,48 +39,30 @@ function personality_modules
clear_personality_queue
- case ${testtype} in
- javac)
- personality_enqueue_module . -DskipTests
- return
- ;;
- mvninstall)
- extra="-DskipTests -DHBasePatchProcess"
- if [[ ${repostatus} == branch ]]; then
- personality_enqueue_module . "${extra}"
- return
+ extra="-DHBasePatchProcess"
+
+ if [[ ${repostatus} == branch
+ && ${testtype} == mvninstall ]];then
+ personality_enqueue_module . ${extra}
+ return
+ fi
+
+ if [[ ${testtype} = findbugs ]]; then
+ for module in ${CHANGED_MODULES}; do
+ # skip findbugs on hbase-shell
+ if [[ ${module} == hbase-shell ]]; then
+ continue
+ else
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module} ${extra}
fi
- return
- ;;
- asflicense)
- # this is very fast and provides the full path if we do it from
- # the root of the source
- personality_enqueue_module . -DHBasePatchProcess
- return
- ;;
- unit)
- if [[ ${TEST_PARALLEL} == "true" ]] ; then
- extra="-Pparallel-tests -DHBasePatchProcess"
- if [[ -n ${TEST_THREADS:-} ]]; then
- extra="${extra} -DtestsThreadCount=${TEST_THREADS}"
- fi
- fi
- ;;
- *)
- extra="-DskipTests -DHBasePatchProcess"
- ;;
- esac
+ done
+ return
+ fi
for module in ${CHANGED_MODULES}; do
-
- # skip findbugs on hbase-shell
- if [[ ${module} == hbase-shell
- && ${testtype} == findbugs ]]; then
- continue
- else
- # shellcheck disable=SC2086
- personality_enqueue_module ${module} ${extra}
- fi
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module} ${extra}
done
}
@@ -97,25 +79,30 @@ function hadoopcheck_filefilter
fi
}
-function hadoopcheck_postapply
+function hadoopcheck_rebuild
{
- local HADOOP2_VERSION
+ local repostatus=$1
+ local hadoopver
local logfile
local count
local result=0
+ if [[ "${repostatus}" = branch ]]; then
+ return 0
+ fi
+
big_console_header "Compiling against various Hadoop versions"
export MAVEN_OPTS="${MAVEN_OPTS}"
- for HADOOP2_VERSION in ${HADOOP2_VERSIONS}; do
- logfile="${PATCH_DIR}/patch-javac-${HADOOP2_VERSION}.txt"
+ for hadoopver in ${HBASE_HADOOP_VERSIONS}; do
+ logfile="${PATCH_DIR}/patch-javac-${hadoopver}.txt"
echo_and_redirect "${logfile}" \
- "${MVN}" clean install \
+ "${MAVEN}" clean install \
-DskipTests -DHBasePatchProcess \
- -Dhadoop-two.version="${HADOOP2_VERSION}"
+ -Dhadoop-two.version="${hadoopver}"
count=$(${GREP} -c ERROR "${logfile}")
if [[ ${count} -gt 0 ]]; then
- add_vote_table -1 hadoopcheck "Patch causes ${count} errors with Hadoop v${HADOOP2_VERSION}."
+ add_vote_table -1 hadoopcheck "Patch causes ${count} errors with Hadoop v${hadoopver}."
((result=result+1))
fi
done
@@ -124,7 +111,7 @@ function hadoopcheck_postapply
return 1
fi
- add_vote_table +1 hadoopcheck "Patch does not cause any errors with Hadoop ${HADOOP2_VERSIONS}."
+ add_vote_table +1 hadoopcheck "Patch does not cause any errors with Hadoop ${HBASE_HADOOP_VERSIONS}."
return 0
}
@@ -141,7 +128,7 @@ function hbaseprotoc_filefilter
fi
}
-function hbaseprotoc_postapply
+function hbaseprotoc_rebuild
{
local i=0
local fn
@@ -150,16 +137,20 @@ function hbaseprotoc_postapply
local count
local result
- big_console_header "Patch HBase protoc plugin"
-
- start_clock
+ if [[ "${repostatus}" = branch ]]; then
+ return 0
+ fi
verify_needed_test hbaseprotoc
if [[ $? == 0 ]]; then
- echo "Patch does not need hbaseprotoc testing."
return 0
fi
+ big_console_header "Patch HBase protoc plugin"
+
+ start_clock
+
+
personality_modules patch hbaseprotoc
modules_workers patch hbaseprotoc compile -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess
@@ -204,28 +195,28 @@ function hbaseanti_filefilter
fi
}
-function hbaseanti_preapply
+function hbaseanti_patchfile
{
+ local patchfile=$1
local warnings
local result
- big_console_header "Checking for known anti-patterns"
-
- start_clock
-
verify_needed_test hbaseanti
if [[ $? == 0 ]]; then
- echo "Patch does not need hbaseanti testing."
return 0
fi
- warnings=$(${GREP} 'new TreeMap<byte.*()' "${PATCH_DIR}/patch")
+ big_console_header "Checking for known anti-patterns"
+
+ start_clock
+
+ warnings=$(${GREP} 'new TreeMap<byte.*()' "${patchfile}")
if [[ ${warnings} -gt 0 ]]; then
add_vote_table -1 hbaseanti "" "The patch appears to have anti-pattern where BYTES_COMPARATOR was omitted: ${warnings}."
((result=result+1))
fi
- warnings=$(${GREP} 'import org.apache.hadoop.classification' "${PATCH_DIR}/patch")
+ warnings=$(${GREP} 'import org.apache.hadoop.classification' "${patchfile}")
if [[ ${warnings} -gt 0 ]]; then
add_vote_table -1 hbaseanti "" "The patch appears use Hadoop classification instead of HBase: ${warnings}."
((result=result+1))
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/kafka.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/kafka.sh b/dev-support/personality/kafka.sh
new file mode 100755
index 0000000..83e5313
--- /dev/null
+++ b/dev-support/personality/kafka.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#shellcheck disable=SC2034
+PATCH_BRANCH_DEFAULT=trunk
+#shellcheck disable=SC2034
+JIRA_ISSUE_RE='^KAFKA-[0-9]+$'
+#shellcheck disable=SC2034
+HOW_TO_CONTRIBUTE="http://kafka.apache.org/contributing.html"
+# shellcheck disable=SC2034
+BUILDTOOL=gradle
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/kafka"
+
+function personality_modules
+{
+ declare repostatus=$1
+ declare testtype=$2
+ declare module
+ declare extra=""
+
+ yetus_debug "Using kafka personality_modules"
+ yetus_debug "Personality: ${repostatus} ${testtype}"
+
+ clear_personality_queue
+
+ case ${testtype} in
+ gradleboot)
+ # kafka's bootstrap is broken
+ if [[ ${testtype} == gradleboot ]]; then
+ pushd "${BASEDIR}" >/dev/null
+ echo_and_redirect "${PATCH_DIR}/kafka-configure-gradle.txt" gradle
+ popd >/dev/null
+ fi
+ ;;
+ compile)
+ extra="clean jar"
+ ;;
+ esac
+
+ for module in ${CHANGED_MODULES}; do
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module} ${extra}
+ done
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/pig.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/pig.sh b/dev-support/personality/pig.sh
index d67b227..91c29f8 100755
--- a/dev-support/personality/pig.sh
+++ b/dev-support/personality/pig.sh
@@ -43,7 +43,7 @@ function personality_modules
ANT_FINDBUGSXML="${BASEDIR}/build/test/findbugs/pig-findbugs-report.xml"
extra="-Dfindbugs.home=${FINDBUGS_HOME}"
;;
- javac)
+ compile)
extra="${extra} -Djavac.args=-Xlint -Dcompile.c++=yes clean piggybank"
;;
javadoc)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/samza.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/samza.sh b/dev-support/personality/samza.sh
new file mode 100755
index 0000000..9d39d92
--- /dev/null
+++ b/dev-support/personality/samza.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#shellcheck disable=SC2034
+PATCH_BRANCH_DEFAULT=master
+#shellcheck disable=SC2034
+JIRA_ISSUE_RE='^SAMZA-[0-9]+$'
+#shellcheck disable=SC2034
+HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/SAMZA/Contributor's+Corner"
+# shellcheck disable=SC2034
+BUILDTOOL=gradle
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/samza"
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/tajo.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/tajo.sh b/dev-support/personality/tajo.sh
index 7e7ea97..209d75e 100755
--- a/dev-support/personality/tajo.sh
+++ b/dev-support/personality/tajo.sh
@@ -21,42 +21,4 @@ JIRA_ISSUE_RE='^TAJO-[0-9]+$'
#shellcheck disable=SC2034
GITHUB_REPO="apache/tajo"
#shellcheck disable=SC2034
-HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TAJO/How+to+Contribute+to+Tajo"
-
-function personality_modules
-{
- local repostatus=$1
- local testtype=$2
- local extra=""
-
- yetus_debug "Personality: ${repostatus} ${testtype}"
-
- clear_personality_queue
-
- case ${testtype} in
- mvninstall)
- extra="-DskipTests"
- if [[ ${repostatus} == branch ]]; then
- personality_enqueue_module . "${extra}"
- return
- fi
- return
- ;;
- asflicense)
- # this is very fast and provides the full path if we do it from
- # the root of the source
- personality_enqueue_module .
- return
- ;;
- unit)
- ;;
- *)
- extra="-DskipTests"
- ;;
- esac
-
- for module in ${CHANGED_MODULES}; do
- # shellcheck disable=SC2086
- personality_enqueue_module ${module} ${extra}
- done
-}
+HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TAJO/How+to+Contribute+to+Tajo"
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/personality/tez.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/tez.sh b/dev-support/personality/tez.sh
index 9b45759..39f0b53 100755
--- a/dev-support/personality/tez.sh
+++ b/dev-support/personality/tez.sh
@@ -22,47 +22,3 @@ JIRA_ISSUE_RE='^TEZ-[0-9]+$'
GITHUB_REPO="apache/tez"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez"
-
-function personality_modules
-{
- local repostatus=$1
- local testtype=$2
- local extra=""
-
- yetus_debug "Personality: ${repostatus} ${testtype}"
-
- clear_personality_queue
-
- case ${testtype} in
- mvninstall)
- extra="-DskipTests"
- if [[ ${repostatus} == branch ]]; then
- personality_enqueue_module . "${extra}"
- return
- fi
- return
- ;;
- asflicense)
- # this is very fast and provides the full path if we do it from
- # the root of the source
- personality_enqueue_module .
- return
- ;;
- unit)
- if [[ ${TEST_PARALLEL} == "true" ]] ; then
- extra="-Pparallel-tests"
- if [[ -n ${TEST_THREADS:-} ]]; then
- extra="${extra} -DtestsThreadCount=${TEST_THREADS}"
- fi
- fi
- ;;
- *)
- extra="-DskipTests"
- ;;
- esac
-
- for module in ${CHANGED_MODULES}; do
- # shellcheck disable=SC2086
- personality_enqueue_module ${module} ${extra}
- done
-}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/ant.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/ant.sh b/dev-support/test-patch.d/ant.sh
new file mode 100755
index 0000000..8d1f364
--- /dev/null
+++ b/dev-support/test-patch.d/ant.sh
@@ -0,0 +1,183 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if [[ -z "${ANT_HOME:-}" ]]; then
+ ANT=ant
+else
+ ANT=${ANT_HOME}/bin/ant
+fi
+
+add_build_tool ant
+
+declare -a ANT_ARGS=("-noinput")
+
+function ant_usage
+{
+ echo "ant specific:"
+ echo "--ant-cmd=<cmd> The 'ant' command to use (default \${ANT_HOME}/bin/ant, or 'ant')"
+}
+
+function ant_parse_args
+{
+ local i
+
+ for i in "$@"; do
+ case ${i} in
+ --ant-cmd=*)
+ ANT=${i#*=}
+ ;;
+ esac
+ done
+
+ # if we requested offline, pass that to ant
+ if [[ ${OFFLINE} == "true" ]]; then
+ ANT_ARGS=("${ANT_ARGS[@]}" -Doffline=)
+ fi
+}
+
+function ant_buildfile
+{
+ echo "build.xml"
+}
+
+function ant_executor
+{
+ echo "${ANT}" "${ANT_ARGS[@]}"
+}
+
+function ant_modules_worker
+{
+ declare repostatus=$1
+ declare tst=$2
+ shift 2
+
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=false
+
+ case ${tst} in
+ findbugs)
+ modules_workers "${repostatus}" findbugs findbugs
+ ;;
+ compile)
+ modules_workers "${repostatus}" compile
+ ;;
+ distclean)
+ modules_workers "${repostatus}" distclean clean
+ ;;
+ javadoc)
+ modules_workers "${repostatus}" javadoc clean javadoc
+ ;;
+ unit)
+ modules_workers "${repostatus}" unit
+ ;;
+ *)
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=true
+ if [[ ${repostatus} = patch ]]; then
+ add_footer_table "${tst}" "not supported by the ${BUILDTOOL} plugin"
+ fi
+ yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}"
+ return 1
+ ;;
+ esac
+}
+
+function ant_javac_count_probs
+{
+ declare warningfile=$1
+ declare val1
+ declare val2
+
+ #shellcheck disable=SC2016
+ val1=$(${GREP} -E "\[javac\] [0-9]+ errors?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}')
+ #shellcheck disable=SC2016
+ val2=$(${GREP} -E "\[javac\] [0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}')
+ echo $((val1+val2))
+}
+
+## @description Helper for check_patch_javadoc
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function ant_javadoc_count_probs
+{
+ local warningfile=$1
+ local val1
+ local val2
+
+ #shellcheck disable=SC2016
+ val1=$(${GREP} -E "\[javadoc\] [0-9]+ errors?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}')
+ #shellcheck disable=SC2016
+ val2=$(${GREP} -E "\[javadoc\] [0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$2} END {print sum}')
+ echo $((val1+val2))
+}
+
+function ant_builtin_personality_modules
+{
+ local repostatus=$1
+ local testtype=$2
+
+ local module
+
+ yetus_debug "Using builtin personality_modules"
+ yetus_debug "Personality: ${repostatus} ${testtype}"
+
+ clear_personality_queue
+
+ for module in ${CHANGED_MODULES}; do
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module}
+ done
+}
+
+function ant_builtin_personality_file_tests
+{
+ local filename=$1
+
+ yetus_debug "Using builtin ant personality_file_tests"
+
+ if [[ ${filename} =~ \.sh
+ || ${filename} =~ \.cmd
+ ]]; then
+ yetus_debug "tests/shell: ${filename}"
+ elif [[ ${filename} =~ \.c$
+ || ${filename} =~ \.cc$
+ || ${filename} =~ \.h$
+ || ${filename} =~ \.hh$
+ || ${filename} =~ \.proto$
+ || ${filename} =~ src/test
+ || ${filename} =~ \.cmake$
+ || ${filename} =~ CMakeLists.txt
+ ]]; then
+ yetus_debug "tests/units: ${filename}"
+ add_test javac
+ add_test unit
+ elif [[ ${filename} =~ build.xml
+ || ${filename} =~ ivy.xml
+ || ${filename} =~ \.java$
+ ]]; then
+ yetus_debug "tests/javadoc+units: ${filename}"
+ add_test javac
+ add_test javadoc
+ add_test unit
+ fi
+
+ if [[ ${filename} =~ \.java$ ]]; then
+ add_test findbugs
+ fi
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/asflicense.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/asflicense.sh b/dev-support/test-patch.d/asflicense.sh
index 27349bf..d839d5f 100755
--- a/dev-support/test-patch.d/asflicense.sh
+++ b/dev-support/test-patch.d/asflicense.sh
@@ -14,11 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-if [[ ${BUILDTOOL} == maven
- || ${BUILDTOOL} == ant ]]; then
- add_plugin asflicense
- add_test asflicense
-fi
+
+add_plugin asflicense
+add_test asflicense
## @description Verify all files have an Apache License
## @audience private
@@ -26,7 +24,7 @@ fi
## @replaceable no
## @return 0 on success
## @return 1 on failure
-function asflicense_postapply
+function asflicense_tests
{
local numpatch
@@ -36,12 +34,15 @@ function asflicense_postapply
personality_modules patch asflicense
case ${BUILDTOOL} in
- maven)
- modules_workers patch asflicense apache-rat:check
- ;;
ant)
modules_workers patch asflicense releaseaudit
;;
+ gradle)
+ modules_workers patch asflicense rat
+ ;;
+ maven)
+ modules_workers patch asflicense apache-rat:check
+ ;;
*)
return 0
;;
@@ -55,7 +56,9 @@ function asflicense_postapply
fi
#shellcheck disable=SC2038
- find "${BASEDIR}" -name rat.txt -o -name releaseaudit_report.txt \
+ find "${BASEDIR}" -name rat.txt \
+ -o -name releaseaudit_report.txt \
+ -o -name rat-report.txt \
| xargs cat > "${PATCH_DIR}/patch-asflicense.txt"
if [[ -s "${PATCH_DIR}/patch-asflicense.txt" ]] ; then
@@ -77,7 +80,7 @@ function asflicense_postapply
add_footer_table asflicense "@@BASE@@/patch-asflicense-problems.txt"
fi
else
- # if we're here, then maven actually failed
+ # if we're here, then build actually failed
modules_messages patch asflicense true
fi
return 1
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/author.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/author.sh b/dev-support/test-patch.d/author.sh
new file mode 100755
index 0000000..49824a5
--- /dev/null
+++ b/dev-support/test-patch.d/author.sh
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+add_plugin author
+
+## @description Check the current directory for @author tags
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function author_patchfile
+{
+ declare patchfile=$1
+ declare authorTags
+ # shellcheck disable=SC2155
+ declare -r appname=$(basename "${BASH_SOURCE-$0}")
+
+ big_console_header "Checking there are no @author tags in the patch."
+
+ start_clock
+
+ if [[ ${CHANGED_FILES} =~ ${appname} ]]; then
+ echo "Skipping @author checks as ${appname} has been patched."
+ add_vote_table 0 @author "Skipping @author checks as ${appname} has been patched."
+ return 0
+ fi
+
+ authorTags=$("${GREP}" -c -i '^[^-].*@author' "${patchfile}")
+ echo "There appear to be ${authorTags} @author tags in the patch."
+ if [[ ${authorTags} != 0 ]] ; then
+ add_vote_table -1 @author \
+ "The patch appears to contain ${authorTags} @author tags which the" \
+ " community has agreed to not allow in code contributions."
+ return 1
+ fi
+ add_vote_table +1 @author "The patch does not contain any @author tags."
+ return 0
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/builtin-bugsystem.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/builtin-bugsystem.sh b/dev-support/test-patch.d/builtin-bugsystem.sh
old mode 100644
new mode 100755
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/builtin-personality.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/builtin-personality.sh b/dev-support/test-patch.d/builtin-personality.sh
index 4be3bfa..1f96e03 100755
--- a/dev-support/test-patch.d/builtin-personality.sh
+++ b/dev-support/test-patch.d/builtin-personality.sh
@@ -14,148 +14,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-function builtin_personality_modules
-{
- local repostatus=$1
- local testtype=$2
-
- local module
-
- yetus_debug "Using builtin personality_modules"
- yetus_debug "Personality: ${repostatus} ${testtype}"
-
- clear_personality_queue
-
- # this always makes sure the local repo has a fresh
- # copy of everything per pom rules.
- if [[ ${repostatus} == branch
- && ${testtype} == mvninstall ]];then
- personality_enqueue_module .
- return
- fi
-
- for module in ${CHANGED_MODULES}; do
- # shellcheck disable=SC2086
- personality_enqueue_module ${module}
- done
-}
-
function personality_modules
{
- builtin_personality_modules "$@"
-}
-
-function builtin_mvn_personality_file_tests
-{
- local filename=$1
-
- yetus_debug "Using builtin mvn personality_file_tests"
-
- if [[ ${filename} =~ src/main/webapp ]]; then
- yetus_debug "tests/webapp: ${filename}"
- elif [[ ${filename} =~ \.sh
- || ${filename} =~ \.cmd
- || ${filename} =~ src/main/scripts
- || ${filename} =~ src/test/scripts
- ]]; then
- yetus_debug "tests/shell: ${filename}"
- elif [[ ${filename} =~ \.md$
- || ${filename} =~ \.md\.vm$
- || ${filename} =~ src/site
- || ${filename} =~ src/main/docs
- ]]; then
- yetus_debug "tests/site: ${filename}"
- add_test site
- elif [[ ${filename} =~ \.c$
- || ${filename} =~ \.cc$
- || ${filename} =~ \.h$
- || ${filename} =~ \.hh$
- || ${filename} =~ \.proto$
- || ${filename} =~ \.cmake$
- || ${filename} =~ CMakeLists.txt
- ]]; then
- yetus_debug "tests/units: ${filename}"
- add_test cc
- add_test unit
- elif [[ ${filename} =~ \.scala$ ]]; then
- add_test javac
- add_test unit
- add_test mvninstall
- elif [[ ${filename} =~ build.xml$
- || ${filename} =~ pom.xml$
- || ${filename} =~ \.java$
- || ${filename} =~ src/main
- ]]; then
- yetus_debug "tests/javadoc+units: ${filename}"
- add_test javac
- add_test javadoc
- add_test mvninstall
- add_test unit
- fi
-
- if [[ ${filename} =~ src/test ]]; then
- yetus_debug "tests"
- add_test unit
- fi
-
- if [[ ${filename} =~ \.java$ ]]; then
- add_test findbugs
- fi
-}
-
-function builtin_ant_personality_file_tests
-{
- local filename=$1
-
- yetus_debug "Using builtin ant personality_file_tests"
-
- if [[ ${filename} =~ \.sh
- || ${filename} =~ \.cmd
- ]]; then
- yetus_debug "tests/shell: ${filename}"
- elif [[ ${filename} =~ \.c$
- || ${filename} =~ \.cc$
- || ${filename} =~ \.h$
- || ${filename} =~ \.hh$
- || ${filename} =~ \.proto$
- || ${filename} =~ src/test
- || ${filename} =~ \.cmake$
- || ${filename} =~ CMakeLists.txt
- ]]; then
- yetus_debug "tests/units: ${filename}"
- add_test javac
- add_test unit
- elif [[ ${filename} =~ build.xml
- || ${filename} =~ ivy.xml
- || ${filename} =~ \.java$
- ]]; then
- yetus_debug "tests/javadoc+units: ${filename}"
- add_test javac
- add_test javadoc
- add_test unit
- fi
-
- if [[ ${filename} =~ \.java$ ]]; then
- add_test findbugs
- fi
-}
-
-function builtin_personality_file_tests
-{
- case ${BUILDTOOL} in
- maven)
- builtin_mvn_personality_file_tests "$@"
- ;;
- ant)
- builtin_ant_personality_file_tests "$@"
- ;;
- *)
- return 1
- ;;
- esac
+ "${BUILDTOOL}_builtin_personality_modules" "$@"
}
function personality_file_tests
{
- builtin_personality_file_tests "$@"
+ "${BUILDTOOL}_builtin_personality_file_tests" "$@"
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/checkstyle.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/checkstyle.sh b/dev-support/test-patch.d/checkstyle.sh
index 5538790..bc47b57 100755
--- a/dev-support/test-patch.d/checkstyle.sh
+++ b/dev-support/test-patch.d/checkstyle.sh
@@ -60,21 +60,28 @@ function checkstyle_runner
modulesuffix=$(basename "${MODULE[${i}]}")
output="${PATCH_DIR}/${repostatus}-checkstyle-${fn}.txt"
logfile="${PATCH_DIR}/maven-${repostatus}-checkstyle-${fn}.txt"
- pushd "${BASEDIR}/${MODULE[${i}]}" >/dev/null
+
+ if [[ ${BUILDTOOLCWD} == true ]]; then
+ pushd "${BASEDIR}/${MODULE[${i}]}" >/dev/null
+ fi
case ${BUILDTOOL} in
- maven)
- cmd="${MVN} ${MAVEN_ARGS[*]} \
- checkstyle:checkstyle \
- -Dcheckstyle.consoleOutput=true \
- ${MODULEEXTRAPARAM[${i}]//@@@MODULEFN@@@/${fn}} -Ptest-patch"
- ;;
ant)
cmd="${ANT} \
-Dcheckstyle.consoleOutput=true \
${MODULEEXTRAPARAM[${i}]//@@@MODULEFN@@@/${fn}} \
${ANT_ARGS[*]} checkstyle"
;;
+ maven)
+ cmd="${MAVEN} ${MAVEN_ARGS[*]} \
+ checkstyle:checkstyle \
+ -Dcheckstyle.consoleOutput=true \
+ ${MODULEEXTRAPARAM[${i}]//@@@MODULEFN@@@/${fn}} -Ptest-patch"
+ ;;
+ *)
+ UNSUPPORTED_TEST=true
+ return 0
+ ;;
esac
#shellcheck disable=SC2086
@@ -101,8 +108,10 @@ function checkstyle_runner
done
rm "${tmp}" 2>/dev/null
- # shellcheck disable=SC2086
- popd >/dev/null
+
+ if [[ ${BUILDTOOLCWD} == true ]]; then
+ popd >/dev/null
+ fi
((i=i+1))
done
@@ -114,20 +123,30 @@ function checkstyle_runner
return 0
}
-function checkstyle_preapply
+function checkstyle_postcompile
{
- local result
+ declare repostatus=$1
- big_console_header "${PATCH_BRANCH} checkstyle"
+ if [[ "${repostatus}" = branch ]]; then
+ checkstyle_preapply
+ else
+ checkstyle_postapply
+ fi
+}
- start_clock
+function checkstyle_preapply
+{
+ local result
verify_needed_test checkstyle
if [[ $? == 0 ]]; then
- echo "Patch does not need checkstyle testing."
return 0
fi
+ big_console_header "${PATCH_BRANCH} checkstyle"
+
+ start_clock
+
personality_modules branch checkstyle
checkstyle_runner branch
result=$?
@@ -151,20 +170,22 @@ function checkstyle_postapply
local numpostpatch=0
local diffpostpatch=0
- big_console_header "Patch checkstyle plugin"
-
- start_clock
-
verify_needed_test checkstyle
if [[ $? == 0 ]]; then
- echo "Patch does not need checkstyle testing."
return 0
fi
+ big_console_header "Patch checkstyle plugin"
+
+ start_clock
+
personality_modules patch checkstyle
checkstyle_runner patch
result=$?
+ if [[ ${UNSUPPORTED_TEST} = true ]]; then
+ return 0
+ fi
# add our previous elapsed to our new timer
# by setting the clock back
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/findbugs.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/findbugs.sh b/dev-support/test-patch.d/findbugs.sh
index df479af..cc251bb 100755
--- a/dev-support/test-patch.d/findbugs.sh
+++ b/dev-support/test-patch.d/findbugs.sh
@@ -89,14 +89,11 @@ function findbugs_runner
local savestop
personality_modules "${name}" findbugs
- case ${BUILDTOOL} in
- maven)
- modules_workers "${name}" findbugs test-compile findbugs:findbugs
- ;;
- ant)
- modules_workers "${name}" findbugs findbugs
- ;;
- esac
+ "${BUILDTOOL}_modules_worker" "${name}" findbugs
+
+ if [[ ${UNSUPPORTED_TEST} = true ]]; then
+ return 0
+ fi
#shellcheck disable=SC2153
until [[ ${i} -eq ${#MODULE[@]} ]]; do
@@ -119,6 +116,7 @@ function findbugs_runner
;;
esac
+
if [[ ! -f ${file} ]]; then
module_status ${i} -1 "" "${name}/${module} no findbugs output file (${file})"
((i=i+1))
@@ -185,12 +183,9 @@ function findbugs_preapply
local module_findbugs_warnings
local result=0
- big_console_header "Pre-patch findbugs detection"
-
verify_needed_test findbugs
if [[ $? == 0 ]]; then
- echo "Patch does not appear to need findbugs tests."
return 0
fi
@@ -199,9 +194,15 @@ function findbugs_preapply
return 1
fi
+ big_console_header "Pre-patch findbugs detection"
+
findbugs_runner branch
result=$?
+ if [[ ${UNSUPPORTED_TEST} = true ]]; then
+ return 0
+ fi
+
if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" ]]; then
until [[ $i -eq ${#MODULE[@]} ]]; do
if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then
@@ -262,15 +263,14 @@ function findbugs_postinstall
local result=0
local savestop
- big_console_header "Patch findbugs detection"
-
verify_needed_test findbugs
if [[ $? == 0 ]]; then
- echo "Patch does not appear to need findbugs tests."
return 0
fi
+ big_console_header "Patch findbugs detection"
+
findbugs_is_installed
if [[ $? != 0 ]]; then
return 1
@@ -278,6 +278,10 @@ function findbugs_postinstall
findbugs_runner patch
+ if [[ ${UNSUPPORTED_TEST} = true ]]; then
+ return 0
+ fi
+
until [[ $i -eq ${#MODULE[@]} ]]; do
if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then
((result=result+1))
@@ -287,7 +291,11 @@ function findbugs_postinstall
start_clock
offset_clock "${MODULE_STATUS_TIMER[${i}]}"
module="${MODULE[${i}]}"
- pushd "${module}" >/dev/null
+
+ if [[ ${BUILDTOOLCWD} == true ]]; then
+ pushd "${module}" >/dev/null
+ fi
+
fn=$(module_file_fragment "${module}")
combined_xml="${PATCH_DIR}/combined-findbugs-${fn}.xml"
@@ -306,7 +314,9 @@ function findbugs_postinstall
"${branchxml}" \
"${patchxml}"
if [[ $? != 0 ]]; then
- popd >/dev/null
+ if [[ ${BUILDTOOLCWD} == true ]]; then
+ popd >/dev/null
+ fi
module_status ${i} -1 "" "${module} cannot run computeBugHistory from findbugs"
((result=result+1))
savestop=$(stop_clock)
@@ -379,3 +389,14 @@ function findbugs_postinstall
fi
return 0
}
+
+function findbugs_rebuild
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ findbugs_preapply
+ else
+ findbugs_postinstall
+ fi
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/gradle.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/gradle.sh b/dev-support/test-patch.d/gradle.sh
new file mode 100755
index 0000000..353eb91
--- /dev/null
+++ b/dev-support/test-patch.d/gradle.sh
@@ -0,0 +1,248 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+add_build_tool gradle
+
+declare -a GRADLE_ARGS=()
+
+function gradle_usage
+{
+ echo "gradle specific:"
+ echo "--gradle-cmd=<cmd> The 'gradle' command to use (default 'gradle')"
+ echo "--gradlew-cmd=<cmd> The 'gradle' command to use (default 'basedir/gradlew')"
+}
+
+function gradle_parse_args
+{
+ local i
+
+ for i in "$@"; do
+ case ${i} in
+ --gradle-cmd=*)
+ GRADLE=${i#*=}
+ ;;
+ --gradlew-cmd=*)
+ GRADLEW=${i#*=}
+ ;;
+ esac
+ done
+
+ # if we requested offline, pass that to mvn
+ if [[ ${OFFLINE} == "true" ]]; then
+ GRADLE_ARGS=("${GRADLE_ARGS[@]}" --offline)
+ fi
+
+ GRADLE=${GRADLE:-gradle}
+ GRADLEW=${GRADLEW:-"${BASEDIR}/gradlew"}
+}
+
+function gradle_initialize
+{
+ if [[ "${BUILDTOOL}" = gradle ]]; then
+ # shellcheck disable=SC2034
+ BUILDTOOLCWD=false
+ fi
+}
+
+function gradle_buildfile
+{
+ echo "gradlew"
+}
+
+function gradle_executor
+{
+ echo "${GRADLEW}" "${GRADLE_ARGS[@]}"
+}
+
+## @description Bootstrap gradle
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function gradle_precompile
+{
+ declare repostatus=$1
+ declare result=0
+
+ if [[ ${BUILDTOOL} != gradle ]]; then
+ return 0
+ fi
+
+ if [[ "${repostatus}" = branch ]]; then
+ # shellcheck disable=SC2153
+ big_console_header "${PATCH_BRANCH} gradle bootstrap"
+ else
+ big_console_header "Patch gradle bootstrap"
+ fi
+
+ personality_modules "${repostatus}" gradleboot
+
+ pushd "${BASEDIR}" >/dev/null
+ echo_and_redirect "${PATCH_DIR}/${repostatus}-gradle-bootstrap.txt" gradle -b bootstrap.gradle
+ popd >/dev/null
+
+ modules_workers "${repostatus}" gradleboot
+ result=$?
+ modules_messages "${repostatus}" gradleboot true
+ if [[ ${result} != 0 ]]; then
+ return 1
+ fi
+ return 0
+}
+
+function gradle_scalac_count_probs
+{
+ local warningfile=$1
+
+ #shellcheck disable=SC2016,SC2046
+ ${GREP} "^/.*.scala:[0-9]*:" "${warningfile}" | wc -l
+}
+
+function gradle_javac_count_probs
+{
+ echo 0
+}
+
+function gradle_javadoc_count_probs
+{
+ echo 0
+}
+
+## @description Helper for check_patch_javadoc
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function gradle_scaladoc_count_probs
+{
+ local warningfile=$1
+
+ #shellcheck disable=SC2016,SC2046
+ ${GREP} "^\[ant:scaladoc\]" "${warningfile}" | wc -l
+}
+
+function gradle_modules_worker
+{
+ declare repostatus=$1
+ declare tst=$2
+ shift 2
+
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=false
+
+ case ${tst} in
+ checkstyle)
+ modules_workers "${repostatus}" "${tst}" checkstyleMain checkstyleTest
+ ;;
+ compile)
+ modules_workers "${repostatus}" "${tst}"
+ ;;
+ distclean)
+ modules_workers "${repostatus}" clean
+ ;;
+ javadoc)
+ modules_workers "${repostatus}" "${tst}" javadoc
+ ;;
+ scaladoc)
+ modules_workers "${repostatus}" "${tst}" scaladoc
+ ;;
+ unit)
+ modules_workers "${repostatus}" "${tst}" test
+ ;;
+ *)
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=true
+ if [[ ${repostatus} = patch ]]; then
+ add_footer_table "${tst}" "not supported by the ${BUILDTOOL} plugin"
+ fi
+ yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}"
+ return 1
+ ;;
+ esac
+}
+
+function gradle_builtin_personality_modules
+{
+ local repostatus=$1
+ local testtype=$2
+
+ local module
+
+ yetus_debug "Using builtin personality_modules"
+ yetus_debug "Personality: ${repostatus} ${testtype}"
+
+ clear_personality_queue
+
+ for module in ${CHANGED_MODULES}; do
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module}
+ done
+}
+
+function gradle_builtin_personality_file_tests
+{
+ local filename=$1
+
+ yetus_debug "Using builtin gradle personality_file_tests"
+
+ if [[ ${filename} =~ src/main/webapp ]]; then
+ yetus_debug "tests/webapp: ${filename}"
+ elif [[ ${filename} =~ \.sh
+ || ${filename} =~ \.cmd
+ || ${filename} =~ src/main/scripts
+ || ${filename} =~ src/test/scripts
+ ]]; then
+ yetus_debug "tests/shell: ${filename}"
+ elif [[ ${filename} =~ \.c$
+ || ${filename} =~ \.cc$
+ || ${filename} =~ \.h$
+ || ${filename} =~ \.hh$
+ || ${filename} =~ \.proto$
+ || ${filename} =~ \.cmake$
+ || ${filename} =~ CMakeLists.txt
+ ]]; then
+ yetus_debug "tests/units: ${filename}"
+ add_test cc
+ add_test unit
+ elif [[ ${filename} =~ \.scala$ ]]; then
+ add_test scalac
+ add_test scaladoc
+ add_test unit
+ elif [[ ${filename} =~ build.xml$
+ || ${filename} =~ pom.xml$
+ || ${filename} =~ \.java$
+ ]]; then
+ yetus_debug "tests/javadoc+units: ${filename}"
+ add_test javac
+ add_test javadoc
+ add_test unit
+ elif [[ ${filename} =~ src/main ]]; then
+ yetus_debug "tests/generic+units: ${filename}"
+ add_test compile
+ add_test unit
+ fi
+
+ if [[ ${filename} =~ src/test ]]; then
+ yetus_debug "tests"
+ add_test unit
+ fi
+
+ if [[ ${filename} =~ \.java$ ]]; then
+ add_test findbugs
+ fi
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/java.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/java.sh b/dev-support/test-patch.d/java.sh
new file mode 100755
index 0000000..b07c67b
--- /dev/null
+++ b/dev-support/test-patch.d/java.sh
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+add_plugin javac
+add_plugin javadoc
+
+function javac_initialize
+{
+ local i
+ local jdkdir
+ local tmplist
+
+ if [[ -z ${JAVA_HOME:-} ]]; then
+ case ${OSTYPE} in
+ Darwin)
+ if [[ -z "${JAVA_HOME}" ]]; then
+ if [[ -x /usr/libexec/java_home ]]; then
+ JAVA_HOME="$(/usr/libexec/java_home)"
+ export JAVA_HOME
+ else
+ export JAVA_HOME=/Library/Java/Home
+ fi
+ fi
+ ;;
+ *)
+ yetus_error "WARNING: JAVA_HOME not defined. Disabling java tests."
+ delete_test javac
+ delete_test javadoc
+ return 1
+ ;;
+ esac
+ fi
+
+ JAVA_HOME=$(cd -P -- "${JAVA_HOME}" >/dev/null && pwd -P)
+
+ for i in ${JDK_DIR_LIST}; do
+ jdkdir=$(cd -P -- "${i}" >/dev/null && pwd -P)
+ if [[ ${jdkdir} != "${JAVA_HOME}" ]]; then
+ tmplist="${tmplist} ${jdkdir}"
+ fi
+ done
+
+ JDK_DIR_LIST="${tmplist} ${JAVA_HOME}"
+ JDK_DIR_LIST=${JDK_DIR_LIST/ }
+}
+
+## @description Verify that ${JAVA_HOME} is defined
+## @audience public
+## @stability stable
+## @replaceable no
+## @return 1 - no JAVA_HOME
+## @return 0 - JAVA_HOME defined
+function javac_precheck
+{
+ declare javaversion
+ declare listofjdks
+ declare i
+
+ start_clock
+
+ if [[ -z ${JAVA_HOME:-} ]]; then
+ yetus_error "ERROR: JAVA_HOME is not defined."
+ add_vote_table -1 pre-patch "JAVA_HOME is not defined."
+ return 1
+ fi
+
+ javaversion=$(report_jvm_version "${JAVA_HOME}")
+ add_footer_table "Default Java" "${javaversion}"
+
+ if [[ -n ${JDK_DIR_LIST}
+ && ${JDK_DIR_LIST} != "${JAVA_HOME}" ]]; then
+ for i in ${JDK_DIR_LIST}; do
+ javaversion=$(report_jvm_version "${i}")
+ listofjdks="${listofjdks} ${i}:${javaversion}"
+ done
+ add_footer_table "Multi-JDK versions" "${listofjdks}"
+ fi
+ return 0
+}
+
+function javac_filefilter
+{
+ declare filename=$1
+
+ if [[ ${filename} =~ \.java$ ]]; then
+ yetus_debug "tests/javac: ${filename}"
+ add_test javac
+ add_test compile
+ fi
+}
+
+function javadoc_filefilter
+{
+ local filename=$1
+
+ if [[ ${filename} =~ \.java$ ]]; then
+ yetus_debug "tests/javadoc: ${filename}"
+ add_test javadoc
+ fi
+}
+
+## @description
+## @audience private
+## @stability stable
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function javac_compile
+{
+ declare codebase=$1
+ declare multijdkmode=$2
+
+ verify_needed_test javac
+ if [[ $? = 0 ]]; then
+ return 0
+ fi
+
+ if [[ ${codebase} = patch ]]; then
+ yetus_debug "javac: calling generic_postlog_compare compile javac ${multijdkmode}"
+ generic_postlog_compare compile javac "${multijdkmode}"
+ fi
+}
+
+## @description Count and compare the number of JavaDoc warnings pre- and post- patch
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function javadoc_rebuild
+{
+ declare codebase=$1
+ declare multijdkmode
+
+ verify_multijdk_test javadoc
+ if [[ $? == 1 ]]; then
+ multijdkmode=true
+ fi
+
+ if [[ "${codebase}" = branch ]]; then
+ generic_pre_handler javadoc "${multijdkmode}"
+ else
+ generic_post_handler javadoc javadoc "${multijdkmode}" true
+ fi
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/jira.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/jira.sh b/dev-support/test-patch.d/jira.sh
index fc649e7..59baf0f 100755
--- a/dev-support/test-patch.d/jira.sh
+++ b/dev-support/test-patch.d/jira.sh
@@ -248,7 +248,7 @@ function jira_determine_branch
count="${PATCH_BRANCH//[^.]}"
total=${#count}
((total = total + 1 ))
- until [[ ${total} -eq 1 ]]; do
+ until [[ ${total} -lt 1 ]]; do
PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1-${total} -d. )
yetus_debug "Determine branch: ISSUE-branch[.##] = ${PATCH_BRANCH}"
((total=total-1))
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/maven.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/maven.sh b/dev-support/test-patch.d/maven.sh
new file mode 100755
index 0000000..94ac383
--- /dev/null
+++ b/dev-support/test-patch.d/maven.sh
@@ -0,0 +1,328 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+declare -a MAVEN_ARGS=("--batch-mode")
+
+if [[ -z "${MAVEN_HOME:-}" ]]; then
+ MAVEN=mvn
+else
+ MAVEN=${MAVEN_HOME}/bin/mvn
+fi
+
+add_plugin mvnsite
+add_plugin mvneclipse
+add_build_tool maven
+
+function maven_usage
+{
+ echo "maven specific:"
+ echo "--mvn-cmd=<cmd> The 'mvn' command to use (default \${MAVEN_HOME}/bin/mvn, or 'mvn')"
+}
+
+function maven_parse_args
+{
+ local i
+
+ for i in "$@"; do
+ case ${i} in
+ --mvn-cmd=*)
+ MAVEN=${i#*=}
+ ;;
+ esac
+ done
+
+ if [[ ${OFFLINE} == "true" ]]; then
+ MAVEN_ARGS=("${MAVEN_ARGS[@]}" --offline)
+ fi
+}
+
+function maven_buildfile
+{
+ echo "pom.xml"
+}
+
+function maven_executor
+{
+ echo "${MAVEN}" "${MAVEN_ARGS[@]}"
+}
+
+function mvnsite_filefilter
+{
+ local filename=$1
+
+ if [[ ${BUILDTOOL} = maven ]]; then
+ if [[ ${filename} =~ src/site ]]; then
+ yetus_debug "tests/mvnsite: ${filename}"
+ add_test mvnsite
+ fi
+ fi
+}
+
+function maven_modules_worker
+{
+ declare repostatus=$1
+ declare tst=$2
+
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=false
+
+ case ${tst} in
+ findbugs)
+ modules_workers "${repostatus}" findbugs test-compile findbugs:findbugs -DskipTests=true
+ ;;
+ compile)
+ modules_workers "${repostatus}" compile clean test-compile -DskipTests=true
+ ;;
+ distclean)
+ modules_workers "${repostatus}" distclean clean -DskipTests=true
+ ;;
+ javadoc)
+ modules_workers "${repostatus}" javadoc clean javadoc:javadoc -DskipTests=true
+ ;;
+ scaladoc)
+ modules_workers "${repostatus}" scaladoc clean scala:doc -DskipTests=true
+ ;;
+ unit)
+ modules_workers "${repostatus}" unit clean test -fae
+ ;;
+ *)
+ # shellcheck disable=SC2034
+ UNSUPPORTED_TEST=true
+ if [[ ${repostatus} = patch ]]; then
+ add_footer_table "${tst}" "not supported by the ${BUILDTOOL} plugin"
+ fi
+ yetus_error "WARNING: ${tst} is unsupported by ${BUILDTOOL}"
+ return 1
+ ;;
+ esac
+}
+
+function maven_javac_count_probs
+{
+ local warningfile=$1
+
+ #shellcheck disable=SC2016,SC2046
+ ${GREP} '\[WARNING\]' "${warningfile}" | ${AWK} '{sum+=1} END {print sum}'
+}
+
+function maven_scalac_count_probs
+{
+ echo 0
+}
+
+## @description Helper for check_patch_javadoc
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function maven_javadoc_count_probs
+{
+ local warningfile=$1
+
+ #shellcheck disable=SC2016,SC2046
+ ${GREP} -E "^[0-9]+ warnings?$" "${warningfile}" | ${AWK} '{sum+=$1} END {print sum}'
+}
+
+function maven_builtin_personality_modules
+{
+ local repostatus=$1
+ local testtype=$2
+
+ local module
+
+ yetus_debug "Using builtin personality_modules"
+ yetus_debug "Personality: ${repostatus} ${testtype}"
+
+ clear_personality_queue
+
+ # this always makes sure the local repo has a fresh
+ # copy of everything per pom rules.
+ if [[ ${repostatus} == branch
+ && ${testtype} == mvninstall ]];then
+ personality_enqueue_module .
+ return
+ fi
+
+ for module in ${CHANGED_MODULES}; do
+ # shellcheck disable=SC2086
+ personality_enqueue_module ${module}
+ done
+}
+
+function maven_builtin_personality_file_tests
+{
+ local filename=$1
+
+ yetus_debug "Using builtin mvn personality_file_tests"
+
+ if [[ ${filename} =~ src/main/webapp ]]; then
+ yetus_debug "tests/webapp: ${filename}"
+ elif [[ ${filename} =~ \.sh
+ || ${filename} =~ \.cmd
+ || ${filename} =~ src/main/scripts
+ || ${filename} =~ src/test/scripts
+ ]]; then
+ yetus_debug "tests/shell: ${filename}"
+ elif [[ ${filename} =~ \.c$
+ || ${filename} =~ \.cc$
+ || ${filename} =~ \.h$
+ || ${filename} =~ \.hh$
+ || ${filename} =~ \.proto$
+ || ${filename} =~ \.cmake$
+ || ${filename} =~ CMakeLists.txt
+ ]]; then
+ yetus_debug "tests/units: ${filename}"
+ add_test cc
+ add_test unit
+ elif [[ ${filename} =~ \.scala$
+ || ${filename} =~ src/scala ]]; then
+ add_test scalac
+ add_test scaladoc
+ add_test unit
+ elif [[ ${filename} =~ build.xml$
+ || ${filename} =~ pom.xml$
+ || ${filename} =~ \.java$
+ || ${filename} =~ src/main
+ ]]; then
+ yetus_debug "tests/javadoc+units: ${filename}"
+ add_test javac
+ add_test javadoc
+ add_test unit
+ fi
+
+ if [[ ${filename} =~ src/test ]]; then
+ yetus_debug "tests"
+ add_test unit
+ fi
+
+ if [[ ${filename} =~ \.java$ ]]; then
+ add_test findbugs
+ fi
+}
+
+## @description Confirm site pre-patch
+## @audience private
+## @stability stable
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function mvnsite_postcompile
+{
+ declare repostatus=$1
+ declare result=0
+
+ if [[ ${BUILDTOOL} != maven ]]; then
+ return 0
+ fi
+
+ verify_needed_test mvnsite
+ if [[ $? == 0 ]];then
+ return 0
+ fi
+
+ if [[ "${repostatus}" = branch ]]; then
+ big_console_header "Pre-patch ${PATCH_BRANCH} maven site verification"
+ else
+ big_console_header "Patch maven site verification"
+ fi
+
+
+ personality_modules "${repostatus}" mvnsite
+ modules_workers "${repostatus}" mvnsite clean site site:stage
+ result=$?
+ modules_messages "${repostatus}" mvnsite true
+ if [[ ${result} != 0 ]]; then
+ return 1
+ fi
+ return 0
+}
+
+## @description Make sure Maven's eclipse generation works.
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function mvneclipse_postcompile
+{
+ declare repostatus=$1
+ declare result=0
+
+ if [[ ${BUILDTOOL} != maven ]]; then
+ return 0
+ fi
+
+ verify_needed_test javac
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+
+ if [[ "${repostatus}" = branch ]]; then
+ big_console_header "Pre-patch ${PATCH_BRANCH} maven eclipse verification"
+ else
+ big_console_header "Patch maven eclipse verification"
+ fi
+
+ personality_modules "${repostatus}" mvneclipse
+ modules_workers "${repostatus}" mvneclipse eclipse:clean eclipse:eclipse
+ result=$?
+ modules_messages "${repostatus}" mvneclipse true
+ if [[ ${result} != 0 ]]; then
+ return 1
+ fi
+ return 0
+}
+
+## @description Verify mvn install works
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success
+## @return 1 on failure
+function maven_precompile
+{
+ declare repostatus=$1
+ declare result=0
+
+ if [[ ${BUILDTOOL} != maven ]]; then
+ return 0
+ fi
+
+ verify_needed_test javadoc
+ result=$?
+
+ verify_needed_test javac
+ ((result = result + $? ))
+ if [[ ${result} == 0 ]]; then
+ return 0
+ fi
+
+ if [[ "${repostatus}" = branch ]]; then
+ big_console_header "Pre-patch ${PATCH_BRANCH} maven install"
+ else
+ big_console_header "Patch maven install"
+ fi
+
+ personality_modules "${repostatus}" mvninstall
+ modules_workers "${repostatus}" mvninstall -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
+ result=$?
+ modules_messages "${repostatus}" mvninstall true
+ if [[ ${result} != 0 ]]; then
+ return 1
+ fi
+ return 0
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/perlcritic.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/perlcritic.sh b/dev-support/test-patch.d/perlcritic.sh
index 1cec3f3..c3ef6c9 100755
--- a/dev-support/test-patch.d/perlcritic.sh
+++ b/dev-support/test-patch.d/perlcritic.sh
@@ -138,3 +138,14 @@ function perlcritic_postapply
add_vote_table +1 perlcritic "There were no new perlcritic issues."
return 0
}
+
+function perlcritic_postcompile
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ perlcritic_preapply
+ else
+ perlcritic_postapply
+ fi
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/pylint.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/pylint.sh b/dev-support/test-patch.d/pylint.sh
index 6fa576e..4519031 100755
--- a/dev-support/test-patch.d/pylint.sh
+++ b/dev-support/test-patch.d/pylint.sh
@@ -166,3 +166,14 @@ function pylint_postapply
add_vote_table +1 pylint "There were no new pylint issues."
return 0
}
+
+function pylint_postcompile
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ pylint_preapply
+ else
+ pylint_postapply
+ fi
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/rubocop.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/rubocop.sh b/dev-support/test-patch.d/rubocop.sh
index 091a2b1..0190289 100755
--- a/dev-support/test-patch.d/rubocop.sh
+++ b/dev-support/test-patch.d/rubocop.sh
@@ -138,3 +138,14 @@ function rubocop_postapply
add_vote_table +1 rubocop "There were no new rubocop issues."
return 0
}
+
+function rubocop_postcompile
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ rubocop_preapply
+ else
+ rubocop_postapply
+ fi
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1e2eeb07/dev-support/test-patch.d/ruby-lint.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/ruby-lint.sh b/dev-support/test-patch.d/ruby-lint.sh
index 35d9604..86b7701 100755
--- a/dev-support/test-patch.d/ruby-lint.sh
+++ b/dev-support/test-patch.d/ruby-lint.sh
@@ -138,3 +138,14 @@ function ruby_lint_postapply
add_vote_table +1 ruby-lint "There were no new ruby-lint issues."
return 0
}
+
+function ruby_lint_postcompile
+{
+ declare repostatus=$1
+
+ if [[ "${repostatus}" = branch ]]; then
+ ruby_lint_preapply
+ else
+ ruby_lint_postapply
+ fi
+}