You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@yetus.apache.org by bu...@apache.org on 2015/09/23 04:24:01 UTC
[31/50] [abbrv] yetus git commit: HADOOP-12129. rework test-patch bug
system support (aw)
HADOOP-12129. rework test-patch bug system support (aw)
Project: http://git-wip-us.apache.org/repos/asf/yetus/repo
Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/463891f3
Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/463891f3
Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/463891f3
Branch: refs/heads/master
Commit: 463891f3fbedb0e3b32749e577dc08dcf29a8152
Parents: 0d65ccd
Author: Allen Wittenauer <aw...@apache.org>
Authored: Mon Aug 17 08:10:35 2015 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Mon Aug 17 08:10:35 2015 -0700
----------------------------------------------------------------------
dev-support/docs/precommit-advanced.md | 59 +-
dev-support/docs/precommit-basic.md | 50 +-
dev-support/personality/flink.sh | 4 +-
dev-support/personality/hadoop.sh | 4 +-
dev-support/personality/hbase.sh | 4 +-
dev-support/personality/pig.sh | 4 +-
dev-support/personality/tajo.sh | 4 +-
dev-support/personality/tez.sh | 4 +-
dev-support/smart-apply-patch.sh | 22 +-
.../test-patch-docker/Dockerfile-startstub | 15 +-
dev-support/test-patch.d/builtin-bugsystem.sh | 163 ++++++
dev-support/test-patch.d/github.sh | 411 +++++++++++++-
dev-support/test-patch.d/jira.sh | 305 +++++++++--
dev-support/test-patch.d/shellcheck.sh | 6 +-
dev-support/test-patch.d/whitespace.sh | 23 +
dev-support/test-patch.sh | 533 +++++++------------
16 files changed, 1174 insertions(+), 437 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/docs/precommit-advanced.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-advanced.md b/dev-support/docs/precommit-advanced.md
index 3185512..7830afe 100644
--- a/dev-support/docs/precommit-advanced.md
+++ b/dev-support/docs/precommit-advanced.md
@@ -52,8 +52,9 @@ test-patch always passes -noinput to Ant. This force ant to be non-interactive.
test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary.
+## Test Plug-ins
-Every plugin must have one line in order to be recognized:
+Every test plugin must have one line in order to be recognized:
```bash
add_plugin <pluginname>
@@ -69,33 +70,69 @@ This function gets called for every file that a patch may contain. This allows
Similarly, there are other functions that may be defined during the test-patch run:
-* pluginname_postcheckout
+* pluginname\_postcheckout
- executed prior to the patch being applied but after the git repository is setup. This is useful for any early error checking that might need to be done before any heavier work.
-* pluginname_preapply
+* pluginname\_preapply
- executed prior to the patch being applied. This is useful for any "before"-type data collection for later comparisons.
-* pluginname_postapply
+* pluginname\_postapply
- executed after the patch has been applied. This is useful for any "after"-type data collection.
-* pluginname_postinstall
+* pluginname\_postinstall
- executed after the mvn install test has been done. If any tests require the Maven repository to be up-to-date with the contents of the patch, this is the place.
-* pluginname_tests
+* pluginname\_tests
- executed after the unit tests have completed.
If the plug-in has some specific options, one can use following functions:
-* pluginname_usage
+* pluginname\_usage
- executed when the help message is displayed. This is used to display the plug-in specific options for the user.
-* pluginname_parse_args
+* pluginname\_parse\_args
- executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
HINT: It is recommended to make the pluginname relatively small, 10 characters at the most. Otherwise, the ASCII output table may be skewed.
+## Bug System Plug-ins
+
+Similar to tests, the ability to add support for bug tracking systems is also handled via a plug-in mechanism.
+
+* pluginname_usage
+
+ - executed when the help message is displayed. This is used to display the plug-in specific options for the user.
+
+* pluginname\_parse\_args
+
+ - executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
+
+
+* pluginname\_locate\_patch
+
+ - Given input from the user, download the patch if possible.
+
+* pluginname\_determine\_branch
+
+ - Using any heuristics available, return the branch to process, if possible.
+
+* pluginname\_determine\_issue
+
+ - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table.
+
+* pluginname\_writecomment
+
+ - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately.
+
+* pluginname\_linecomments
+
+ - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments.
+
+* pluginname\_finalreport
+
+ - Write the final result table to the bug system.
# Configuring for Other Projects
@@ -181,7 +218,7 @@ This function will tell test-patch that when the javadoc test is being run, do t
# Important Variables
-There are a handful of extremely important variables that make life easier for personality and plug-in writers:
+There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information.
* BUILD\_NATIVE will be set to true if the system has requested that non-JVM-based code be built (e.g., JNI or other compiled C code). Under Jenkins, this is always true.
@@ -193,9 +230,11 @@ There are a handful of extremely important variables that make life easier for p
* CHANGED\_MODULES reports which modules that appear to have source code in them.
+* GITHUB\_REPO is to help test-patch when talking to Github. If test-patch is given just a number on the command line, it will default to using this repo to determine the pull request.
+
* HOW\_TO\_CONTRIBUTE should be a URL that points to a project's on-boarding documentation for new users. Currently, it is used to suggest a review of patch naming guidelines. Since this should be project specific information, it is useful to set in a project's personality.
-* ISSUE\_RE is to help test-patch when talking to JIRA. It helps determine if the given project is appropriate for the given JIRA issue.
+* JIRA\_ISSUE\_RE is to help test-patch when talking to JIRA. It helps determine if the given project is appropriate for the given JIRA issue.
* MODULE and other MODULE\_\* are arrays that contain which modules, the status, etc, to be operated upon. These should be treated as read-only by plug-ins.
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/docs/precommit-basic.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-basic.md b/dev-support/docs/precommit-basic.md
index e68ad07..a612214 100644
--- a/dev-support/docs/precommit-basic.md
+++ b/dev-support/docs/precommit-basic.md
@@ -48,9 +48,9 @@ test-patch has the following requirements:
* POSIX awk
* POSIX grep
* POSIX sed
-* wget
+* curl
* file command
-* smart-apply-patch.sh
+* smart-apply-patch.sh (included!)
Maven plugins requirements:
@@ -59,8 +59,8 @@ Maven plugins requirements:
Optional:
-* Apache JIRA-based issue tracking
-* JIRA cli tools
+* JIRA-based issue tracking
+* GitHub-based issue tracking
The locations of these files are (mostly) assumed to be in the file path, but may be overridden via command line options. For Solaris and Solaris-like operating systems, the default location for the POSIX binaries is in /usr/xpg4/bin and the default location for the GNU binaries is /usr/gnu/bin.
@@ -119,6 +119,8 @@ will tell test-patch to use ant instead of maven to drive the project.
# Providing Patch Files
+## JIRA
+
It is a fairly common practice within the Apache community to use Apache's JIRA instance to store potential patches. As a result, test-patch supports providing just a JIRA issue number. test-patch will find the *last* attachment, download it, then process it.
For example:
@@ -129,15 +131,47 @@ $ test-patch.sh (other options) HADOOP-9905
... will process the patch file associated with this JIRA issue.
-A new practice is to use a service such as GitHub and its Pull Request (PR) feature. Luckily, test-patch supports URLs and many services like GitHub provide ways to provide unified diffs via URLs.
+If the Apache JIRA system is not in use, then override options may be provided on the command line to point to a different JIRA instance.
+
+```bash
+$ test-patch.sh --jira-issue-re='^PROJECT-[0-9]+$' --jira-base-url='https://example.com/jira' PROJECT-90
+```
+
+... will process the patch file attached to PROJECT-90 on the JIRA instance located on the example.com server.
+
+## GITHUB
+
+test-patch has some basic support for Github. test-patch supports many forms of providing pull requests to work on:
+
+```bash
+$ test-patch.sh --github-repo=apache/pig 99
+```
+
+or
+
+```bash
+$ test-patch.sh https://github.com/apache/pig/pulls/99
+```
+
+or
+
+```bash
+$ test-patch.sh https://github.com/apache/pig/pulls/99.patch
+```
+
+... will process PR #99 on the apache/pig repo.
+
+## Generic URLs
+
+Luckily, test-patch supports provide ways to provide unified diffs via URLs.
For example:
```bash
-$ test-patch.sh (other options) https://github.com/apache/flink/pull/773.patch
+$ test-patch.sh (other options) https://example.com/webserver/file.patch
```
-... will grab a unified diff of PR #773 and process it.
+... will download and process the file.patch from the example.com webserver.
# Project-specific Capabilities
@@ -181,8 +215,6 @@ $ test-patch.sh (other options) --docker
This will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile, see the advanced guide.
-
-
## In Closing
test-patch has many other features and command line options for the basic user. Many of these are self-explanatory. To see the list of options, run test-patch.sh without any options or with --help.
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/flink.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/flink.sh b/dev-support/personality/flink.sh
index de2a0f1..4b6c390 100755
--- a/dev-support/personality/flink.sh
+++ b/dev-support/personality/flink.sh
@@ -17,7 +17,9 @@
#shellcheck disable=SC2034
PATCH_BRANCH_DEFAULT=master
#shellcheck disable=SC2034
-ISSUE_RE='^FLINK-[0-9]+$'
+JIRA_ISSUE_RE='^FLINK-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/flink"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE=""
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/hadoop.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/hadoop.sh b/dev-support/personality/hadoop.sh
index 1243a17..b3eb04a 100755
--- a/dev-support/personality/hadoop.sh
+++ b/dev-support/personality/hadoop.sh
@@ -21,7 +21,9 @@ PATCH_BRANCH_DEFAULT=trunk
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE="https://wiki.apache.org/hadoop/HowToContribute"
#shellcheck disable=SC2034
-ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$'
+JIRA_ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/hadoop"
#shellcheck disable=SC2034
PYLINT_OPTIONS="--indent-string=' '"
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/hbase.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/hbase.sh b/dev-support/personality/hbase.sh
index 9749096..4f23679 100755
--- a/dev-support/personality/hbase.sh
+++ b/dev-support/personality/hbase.sh
@@ -17,7 +17,9 @@
#shellcheck disable=SC2034
PATCH_BRANCH_DEFAULT=master
#shellcheck disable=SC2034
-ISSUE_RE='^HBASE-[0-9]+$'
+JIRA_ISSUE_RE='^HBASE-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/hbase"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE=""
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/pig.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/pig.sh b/dev-support/personality/pig.sh
index d01a410..d67b227 100755
--- a/dev-support/personality/pig.sh
+++ b/dev-support/personality/pig.sh
@@ -17,7 +17,9 @@
#shellcheck disable=SC2034
PATCH_BRANCH_DEFAULT=trunk
#shellcheck disable=SC2034
-ISSUE_RE='^PIG-[0-9]+$'
+JIRA_ISSUE_RE='^PIG-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/pig"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE=""
#shellcheck disable=SC2034
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/tajo.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/tajo.sh b/dev-support/personality/tajo.sh
index 56e5442..7e7ea97 100755
--- a/dev-support/personality/tajo.sh
+++ b/dev-support/personality/tajo.sh
@@ -17,7 +17,9 @@
#shellcheck disable=SC2034
PATCH_BRANCH_DEFAULT=master
#shellcheck disable=SC2034
-ISSUE_RE='^TAJO-[0-9]+$'
+JIRA_ISSUE_RE='^TAJO-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/tajo"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TAJO/How+to+Contribute+to+Tajo"
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/personality/tez.sh
----------------------------------------------------------------------
diff --git a/dev-support/personality/tez.sh b/dev-support/personality/tez.sh
index 1d6a227..9b45759 100755
--- a/dev-support/personality/tez.sh
+++ b/dev-support/personality/tez.sh
@@ -17,7 +17,9 @@
#shellcheck disable=SC2034
PATCH_BRANCH_DEFAULT=master
#shellcheck disable=SC2034
-ISSUE_RE='^TEZ-[0-9]+$'
+JIRA_ISSUE_RE='^TEZ-[0-9]+$'
+#shellcheck disable=SC2034
+GITHUB_REPO="apache/tez"
#shellcheck disable=SC2034
HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez"
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/smart-apply-patch.sh
----------------------------------------------------------------------
diff --git a/dev-support/smart-apply-patch.sh b/dev-support/smart-apply-patch.sh
index 00e3a0a..e11a734 100755
--- a/dev-support/smart-apply-patch.sh
+++ b/dev-support/smart-apply-patch.sh
@@ -76,7 +76,7 @@ function setup_defaults
SunOS)
AWK=${AWK:-/usr/xpg4/bin/awk}
SED=${SED:-/usr/xpg4/bin/sed}
- WGET=${WGET:-wget}
+ CURL=${CURL:-curl}
GIT=${GIT:-git}
GREP=${GREP:-/usr/xpg4/bin/grep}
PATCH=${PATCH:-/usr/gnu/bin/patch}
@@ -86,7 +86,7 @@ function setup_defaults
*)
AWK=${AWK:-awk}
SED=${SED:-sed}
- WGET=${WGET:-wget}
+ CURL=${CURL:-curl}
GIT=${GIT:-git}
GREP=${GREP:-grep}
PATCH=${PATCH:-patch}
@@ -122,7 +122,7 @@ function yetus_usage
echo "--grep-cmd=<cmd> The 'grep' command to use (default 'grep')"
echo "--git-cmd=<cmd> The 'git' command to use (default 'git')"
echo "--patch-cmd=<cmd> The GNU-compatible 'patch' command to use (default 'patch')"
- echo "--wget-cmd=<cmd> The 'wget' command to use (default 'wget')"
+ echo "--curl-cmd=<cmd> The 'curl' command to use (default 'curl')"
}
## @description Interpret the command line parameters
@@ -162,8 +162,8 @@ function parse_args
--patch-dir=*)
PATCH_DIR=${i#*=}
;;
- --wget-cmd=*)
- WGET=${i#*=}
+ --curl-cmd=*)
+ CURL=${i#*=}
;;
--*)
## PATCH_OR_ISSUE can't be a --. So this is probably
@@ -233,13 +233,15 @@ function locate_patch
echo "Patch is being downloaded at $(date) from"
PATCHURL="${PATCH_OR_ISSUE}"
else
- ${WGET} -q -O "${PATCH_DIR}/jira" "http://issues.apache.org/jira/browse/${PATCH_OR_ISSUE}"
-
+ ${CURL} --silent \
+ --output "${PATCH_DIR}/jira" \
+ --location \
+ "https://issues.apache.org/jira/browse/${PATCH_OR_ISSUE}"
case $? in
0)
;;
2)
- yetus_error "ERROR: .wgetrc/.netrc parsing error."
+ yetus_error "ERROR: .curlrc/.netrc parsing error."
cleanup_and_exit 1
;;
3)
@@ -269,7 +271,7 @@ function locate_patch
#shellcheck disable=SC2016
relativePatchURL=$(${AWK} 'match($0,"\"/jira/secure/attachment/[0-9]*/[^\"]*"){print substr($0,RSTART+1,RLENGTH-1)}' "${PATCH_DIR}/jira" |
${GREP} -v -e 'htm[l]*$' | sort | tail -1)
- PATCHURL="http://issues.apache.org${relativePatchURL}"
+ PATCHURL="https://issues.apache.org${relativePatchURL}"
if [[ ! ${PATCHURL} =~ \.patch$ ]]; then
notSureIfPatch=true
fi
@@ -277,7 +279,7 @@ function locate_patch
fi
fi
if [[ -z "${PATCH_FILE}" ]]; then
- ${WGET} -q -O "${PATCH_DIR}/patch" "${PATCHURL}"
+ ${CURL} --silent --location --output "${PATCH_DIR}/patch" "${PATCHURL}"
if [[ $? != 0 ]];then
yetus_error "ERROR: ${PATCH_OR_ISSUE} could not be downloaded."
cleanup_and_exit 1
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch-docker/Dockerfile-startstub
----------------------------------------------------------------------
diff --git a/dev-support/test-patch-docker/Dockerfile-startstub b/dev-support/test-patch-docker/Dockerfile-startstub
index fd3e4c5..c49b589 100644
--- a/dev-support/test-patch-docker/Dockerfile-startstub
+++ b/dev-support/test-patch-docker/Dockerfile-startstub
@@ -62,8 +62,8 @@ RUN apt-get install -y oracle-java8-installer
# Install findbugs
######
RUN mkdir -p /opt/findbugs && \
- wget https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download \
- -O /opt/findbugs.tar.gz && \
+ curl https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download \
+ -o /opt/findbugs.tar.gz && \
tar xzf /opt/findbugs.tar.gz --strip-components 1 -C /opt/findbugs
ENV FINDBUGS_HOME /opt/findbugs
@@ -83,14 +83,3 @@ RUN gem install rubocop
###
RUN gem install ruby-lint
-#####
-# Install JIRA CLI
-#####
-
-RUN mkdir -p /opt/jiracli && \
- wget https://bobswift.atlassian.net/wiki/download/attachments/16285777/jira-cli-2.2.0-distribution.zip \
- -O /tmp/jiracli.zip && \
- unzip -qq -d /opt/jiracli /tmp/jiracli.zip && \
- ln -s /opt/jiracli/jira-cli-2.2.0 /opt/jiracli/latest && \
- chmod -R a+rx /opt/jiracli/jira-cli-2.2.0
-ENV JIRACLI_HOME /opt/jiracli/latest
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.d/builtin-bugsystem.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/builtin-bugsystem.sh b/dev-support/test-patch.d/builtin-bugsystem.sh
new file mode 100644
index 0000000..9a9ee05
--- /dev/null
+++ b/dev-support/test-patch.d/builtin-bugsystem.sh
@@ -0,0 +1,163 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This bug system handles the output on the screen.
+
+add_bugsystem console
+
+# we always call this one last
+
+function generic_locate_patch
+{
+ declare input=$1
+ declare output=$2
+
+ if [[ "${OFFLINE}" == true ]]; then
+ yetus_debug "generic_locate_patch: offline, skipping"
+ return 1
+ fi
+
+ ${CURL} --silent \
+ --output "${output}" \
+ "${input}"
+ if [[ $? != 0 ]]; then
+ yetus_debug "jira_locate_patch: not a JIRA."
+ return 1
+ fi
+ return 0
+}
+
+## @description Print out the finished details on the console
+## @audience private
+## @stability evolving
+## @replaceable no
+## @param runresult
+## @return 0 on success
+## @return 1 on failure
+function console_finalreport
+{
+ declare result=$1
+ shift
+ declare i=0
+ declare ourstring
+ declare vote
+ declare subs
+ declare ela
+ declare comment
+ declare commentfile1="${PATCH_DIR}/comment.1"
+ declare commentfile2="${PATCH_DIR}/comment.2"
+ declare normaltop
+ declare line
+ declare seccoladj=0
+ declare spcfx=${PATCH_DIR}/spcl.txt
+
+ if [[ ${result} == 0 ]]; then
+ if [[ ${JENKINS} == false ]]; then
+ {
+ printf "IF9fX19fX19fX18gCjwgU3VjY2VzcyEgPgogLS0tLS0tLS0tLSAKIFwgICAg";
+ printf "IC9cICBfX18gIC9cCiAgXCAgIC8vIFwvICAgXC8gXFwKICAgICAoKCAgICBP";
+ printf "IE8gICAgKSkKICAgICAgXFwgLyAgICAgXCAvLwogICAgICAgXC8gIHwgfCAg";
+ printf "XC8gCiAgICAgICAgfCAgfCB8ICB8ICAKICAgICAgICB8ICB8IHwgIHwgIAog";
+ printf "ICAgICAgIHwgICBvICAgfCAgCiAgICAgICAgfCB8ICAgfCB8ICAKICAgICAg";
+ printf "ICB8bXwgICB8bXwgIAo"
+ } > "${spcfx}"
+ fi
+ printf "\n\n+1 overall\n\n"
+ else
+ if [[ ${JENKINS} == false ]]; then
+ {
+ printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf";
+ printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf";
+ printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98";
+ printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg";
+ printf "ICAgICAgICAgICAgICAgICAK"
+ } > "${spcfx}"
+ fi
+ printf "\n\n-1 overall\n\n"
+ fi
+
+ if [[ -f ${spcfx} ]]; then
+ if which base64 >/dev/null 2>&1; then
+ base64 --decode "${spcfx}" 2>/dev/null
+ elif which openssl >/dev/null 2>&1; then
+ openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null
+ fi
+ echo
+ echo
+ rm "${spcfx}"
+ fi
+
+ seccoladj=$(findlargest 2 "${TP_VOTE_TABLE[@]}")
+ if [[ ${seccoladj} -lt 10 ]]; then
+ seccoladj=10
+ fi
+
+ seccoladj=$((seccoladj + 2 ))
+ i=0
+ until [[ $i -eq ${#TP_HEADER[@]} ]]; do
+ printf "%s\n" "${TP_HEADER[${i}]}"
+ ((i=i+1))
+ done
+
+ printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment"
+ echo "============================================================================"
+ i=0
+ until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do
+ ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ')
+ vote=$(echo "${ourstring}" | cut -f2 -d\|)
+ subs=$(echo "${ourstring}" | cut -f3 -d\|)
+ ela=$(echo "${ourstring}" | cut -f4 -d\|)
+ comment=$(echo "${ourstring}" | cut -f5 -d\|)
+
+ echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}"
+ normaltop=$(head -1 "${commentfile1}")
+ ${SED} -e '1d' "${commentfile1}" > "${commentfile2}"
+
+ printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \
+ "${subs}" "${ela}" "${normaltop}"
+ while read line; do
+ printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}"
+ done < "${commentfile2}"
+
+ ((i=i+1))
+ rm "${commentfile2}" "${commentfile1}" 2>/dev/null
+ done
+
+ if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then
+ seccoladj=$(findlargest 1 "${TP_TEST_TABLE[@]}")
+ printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason"
+ i=0
+ until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do
+ ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ')
+ vote=$(echo "${ourstring}" | cut -f2 -d\|)
+ subs=$(echo "${ourstring}" | cut -f3 -d\|)
+ printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}"
+ ((i=i+1))
+ done
+ fi
+
+ printf "\n\n|| Subsystem || Report/Notes ||\n"
+ echo "============================================================================"
+ i=0
+
+ until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do
+ comment=$(echo "${TP_FOOTER_TABLE[${i}]}" |
+ ${SED} -e "s,@@BASE@@,${PATCH_DIR},g")
+ printf "%s\n" "${comment}"
+ ((i=i+1))
+ done
+}
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.d/github.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/github.sh b/dev-support/test-patch.d/github.sh
index 281f15b..36c7e51 100755
--- a/dev-support/test-patch.d/github.sh
+++ b/dev-support/test-patch.d/github.sh
@@ -14,23 +14,403 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+# This bug system provides github integration
+
add_bugsystem github
+# personalities can override the following settings:
+
+# Web interface URL.
+GITHUB_BASE_URL="https://github.com"
+
+# API interface URL.
+GITHUB_API_URL="https://api.github.com"
+
+# user/repo
+GITHUB_REPO=""
+
+# user settings
+GITHUB_PASSWD=""
+GITHUB_TOKEN=""
+GITHUB_USER=""
+GITHUB_ISSUE=""
+
+# private globals...
+GITHUB_BRIDGED=false
+GITHUB_COMMITSHA=""
+
+function github_usage
+{
+ echo "GITHUB Options:"
+ echo "--github-api-url=<url> The URL of the API for github (default: '${GITHUB_API_URL}')"
+ echo "--github-base-url=<url> The URL of the github server (default:'${GITHUB_BASE_URL}')"
+ echo "--github-password=<pw> Github password"
+ echo "--github-repo=<repo> github repo to use (default:'${GITHUB_REPO}')"
+ echo "--github-token=<token> The token to use to write to github"
+ echo "--github-user=<user> Github user"
+}
+
+function github_parse_args
+{
+ declare i
+
+ for i in "$@"; do
+ case ${i} in
+ --github-api-url=*)
+ GITHUB_API_URL=${i#*=}
+ ;;
+ --github-base-url=*)
+ GITHUB_BASE_URL=${i#*=}
+ ;;
+ --github-repo=*)
+ GITHUB_REPO=${i#*=}
+ ;;
+ --github-token=*)
+ GITHUB_TOKEN=${i#*=}
+ ;;
+ --github-password=*)
+ GITHUB_PASSWD=${i#*=}
+ ;;
+ --github-user=*)
+ GITHUB_USER=${i#*=}
+ ;;
+ esac
+ done
+}
+
+## @description this gets called when JIRA thinks this
+## @description issue is just a pointer to github
+## @description WARNING: Called from JIRA plugin!
+function github_jira_bridge
+{
+ declare fileloc=$1
+ declare urlfromjira
+
+ # we use this to prevent loops later on
+ GITHUB_BRIDGED=true
+
+ # the JIRA issue has already been downloaded. So let's
+ # find the URL. This is currently hard-coded to github.com
+ # Sorry Github Enterprise users. :(
+
+ # shellcheck disable=SC2016
+ urlfromjira=$(${AWK} 'match($0,"https://github.com/.*patch"){print $1}' "${PATCH_DIR}/jira" | tail -1)
+ github_breakup_url "${urlfromjira}"
+ github_locate_patch "${GITHUB_ISSUE}" "${fileloc}"
+}
+
+## @description given a URL, break it up into github plugin globals
+## @description this will *override* any personality or yetus defaults
+## @params url
+function github_breakup_url
+{
+ declare url=$1
+ declare count
+ declare pos1
+ declare pos2
+
+ count=${url//[^\/]}
+ count=${#count}
+ ((pos2=count-3))
+ ((pos1=pos2))
+
+ GITHUB_BASE_URL=$(echo "${url}" | cut -f1-${pos2} -d/)
+
+ ((pos1=pos1+1))
+ ((pos2=pos1+1))
+
+ GITHUB_REPO=$(echo "${url}" | cut -f${pos1}-${pos2} -d/)
+
+ ((pos1=pos2+2))
+ unset pos2
+
+ GITHUB_ISSUE=$(echo "${url}" | cut -f${pos1}-${pos2} -d/ | cut -f1 -d.)
+}
+
+
+## @description based upon a github PR, attempt to link back to JIRA
+function github_find_jira_title
+{
+ declare title
+ declare maybe
+ declare retval
+
+ if [[ ! -f "${PATCH_DIR}/github-pull.json" ]]; then
+ return 1
+ fi
+
+ title=$(GREP title "${PATCH_DIR}/github-pull.json" \
+ | cut -f4 -d\")
+
+ # people typically do two types: JIRA-ISSUE: and [JIRA-ISSUE]
+ # JIRA_ISSUE_RE is pretty strict so we need to chop that stuff
+ # out first
+
+ maybe=$(echo "${title}" | cut -f2 -d\[ | cut -f1 -d\])
+ jira_determine_issue "${maybe}"
+ retval=$?
+
+ if [[ ${retval} == 0 ]]; then
+ return 0
+ fi
+
+ maybe=$(echo "${title}" | cut -f1 -d:)
+ jira_determine_issue "${maybe}"
+ retval=$?
+
+ if [[ ${retval} == 0 ]]; then
+ return 0
+ fi
+}
+
+function github_determine_issue
+{
+ declare input=$1
+
+ if [[ ${input} =~ ^[0-9]+$
+ && -n ${GITHUB_REPO} ]]; then
+ # shellcheck disable=SC2034
+ ISSUE=${input}
+ if [[ -z ${GITHUB_ISSUE} ]]; then
+ GITHUB_ISSUE=${input}
+ fi
+ fi
+
+ # if JIRA didn't call us, should we call it?
+ if [[ ${GITHUB_BRIDGED} == false ]]; then
+ github_find_jira_title
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ fi
+
+ if [[ -n ${GITHUB_ISSUE} ]]; then
+ return 0
+ fi
+
+ return 1
+}
+
+## @description Try to guess the branch being tested using a variety of heuristics
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success, with PATCH_BRANCH updated appropriately
+## @return 1 on failure
+function github_determine_branch
+{
+ if [[ ! -f "${PATCH_DIR}/github-pull.json" ]]; then
+ return 1
+ fi
+
+ # shellcheck disable=SC2016
+ PATCH_BRANCH=$(${AWK} 'match($0,"\"ref\": \""){print $2}' "${PATCH_DIR}/github-pull.json"\
+ | cut -f2 -d\"\
+ | tail -1 )
+
+ yetus_debug "Github determine branch: starting with ${PATCH_BRANCH}"
+
+ verify_valid_branch "${PATCH_BRANCH}"
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ return 1
+}
+
+function github_locate_patch
+{
+ declare input=$1
+ declare output=$2
+ declare githubauth
+
+ if [[ "${OFFLINE}" == true ]]; then
+ yetus_debug "github_locate_patch: offline, skipping"
+ return 1
+ fi
+
+
+ # https://github.com/your/repo/pull/##
+ if [[ ${input} =~ ^${GITHUB_BASE_URL}.*/pull/[0-9]+$ ]]; then
+ github_breakup_url "${input}.patch"
+ input=${GITHUB_ISSUE}
+ fi
+
+ # https://github.com/your/repo/pulls/##.patch
+ if [[ ${input} =~ ^${GITHUB_BASE_URL}.*patch$ ]]; then
+ github_breakup_url "${input}"
+ input=${GITHUB_ISSUE}
+ fi
+
+ # https://github.com/your/repo/pulls/##.diff
+ if [[ ${input} =~ ^${GITHUB_BASE_URL}.*diff$ ]]; then
+ github_breakup_url "${input}"
+ input=${GITHUB_ISSUE}
+ fi
+
+ # if it isn't a number at this point, no idea
+ # how to process
+ if [[ ! ${input} =~ ^[0-9]+$ ]]; then
+ yetus_debug "github: ${input} is not a pull request #"
+ return 1
+ fi
+
+ # we always pull the .patch version (even if .diff was given)
+ # with the assumption that this way binary files work.
+ # The downside of this is that the patch files are
+ # significantly larger and therefore take longer to process
+ PATCHURL="${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}.patch"
+ echo "GITHUB PR #${input} is being downloaded at $(date) from"
+ echo "${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}"
+
+ if [[ -n "${GITHUB_USER}"
+ && -n "${GITHUB_PASSWD}" ]]; then
+ githubauth="${GITHUB_USER}:${GITHUB_PASSWD}"
+ elif [[ -n "${GITHUB_TOKEN}" ]]; then
+ githubauth="Authorization: token ${GITHUB_TOKEN}"
+ else
+ githubauth="X-ignore-me: fake"
+ fi
+
+ # Let's pull the PR JSON for later use
+ ${CURL} --silent --fail \
+ -H "Accept: application/vnd.github.v3.full+json" \
+ -H "${githubauth}" \
+ --output "${PATCH_DIR}/github-pull.json" \
+ --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/pulls/${input}"
+
+ echo "Patch from GITHUB PR #${input} is being downloaded at $(date) from"
+ echo "${PATCHURL}"
+
+ # the actual patch file
+ ${CURL} --silent --fail \
+ --output "${output}" \
+ --location \
+ -H "${githubauth}" \
+ "${PATCHURL}"
+
+ if [[ $? != 0 ]]; then
+ yetus_debug "github_locate_patch: not a github pull request."
+ return 1
+ fi
+
+ GITHUB_ISSUE=${input}
+
+ # github will translate this to be #(xx) !
+ add_footer_table "GITHUB PR" "${GITHUB_BASE_URL}/${GITHUB_REPO}/pull/${input}"
+
+ return 0
+}
+
+function github_linecomments
+{
+ declare plugin=$1
+ declare file=$2
+ # shellcheck disable=SC2034
+ declare realline=$3
+ declare uniline=$4
+ declare text=$5
+ declare tempfile="${PATCH_DIR}/ghcomment.$$.${RANDOM}"
+ declare githubauth
+
+ if [[ "${file}" =~ ^./ ]]; then
+ file=${file##./}
+ fi
+
+ if [[ -z "${GITHUB_COMMITSHA}" ]]; then
+ GITHUB_COMMITSHA=$(${GREP} \"sha\" "${PATCH_DIR}/github-pull.json" 2>/dev/null \
+ | head -1 \
+ | cut -f4 -d\")
+ fi
+
+ if [[ -z "${uniline}" ]]; then
+ return
+ fi
+
+ # build our REST post
+ {
+ printf "{\"body\":\""
+ echo "${plugin}: ${text}" \
+ | ${SED} -e 's,\\,\\\\,g' \
+ -e 's,\",\\\",g' \
+ -e 's,$,\\r\\n,g' \
+ | tr -d '\n'
+ echo "\","
+ echo "\"commit_id\":\"${GITHUB_COMMITSHA}\","
+ echo "\"path\":\"${file}\","
+ echo "\"position\":${uniline}"
+ echo "}"
+ } > "${tempfile}"
+
+ if [[ -n "${GITHUB_USER}"
+ && -n "${GITHUB_PASSWD}" ]]; then
+ githubauth="${GITHUB_USER}:${GITHUB_PASSWD}"
+ elif [[ -n "${GITHUB_TOKEN}" ]]; then
+ githubauth="Authorization: token ${GITHUB_TOKEN}"
+ else
+ return 0
+ fi
+
+ ${CURL} -X POST \
+ -H "Accept: application/vnd.github.v3.full+json" \
+ -H "Content-Type: application/json" \
+ -H "${githubauth}" \
+ -d @"${tempfile}" \
+ --silent --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/pulls/${GITHUB_ISSUE}/comments" \
+ >/dev/null
+ rm "${tempfile}"
+}
+
## @description Write the contents of a file to github
## @params filename
## @stability stable
## @audience public
function github_write_comment
{
- local -r commentfile=${1}
- shift
+ declare -r commentfile=${1}
+ declare retval=0
+ declare restfile="${PATCH_DIR}/ghcomment.$$"
+ declare githubauth
- local retval=1
+ if [[ "${OFFLINE}" == true ]]; then
+ echo "Github Plugin: Running in offline, comment skipped."
+ return 0
+ fi
+ {
+ printf "{\"body\":\""
+ ${SED} -e 's,\\,\\\\,g' \
+ -e 's,\",\\\",g' \
+ -e 's,$,\\r\\n,g' "${commentfile}" \
+ | tr -d '\n'
+ echo "\"}"
+ } > "${restfile}"
+
+ if [[ -n "${GITHUB_USER}"
+ && -n "${GITHUB_PASSWD}" ]]; then
+ githubauth="${GITHUB_USER}:${GITHUB_PASSWD}"
+ elif [[ -n "${GITHUB_TOKEN}" ]]; then
+ githubauth="Authorization: token ${GITHUB_TOKEN}"
+ else
+ echo "Github Plugin: no credentials provided to write a comment."
+ return 0
+ fi
+
+ ${CURL} -X POST \
+ -H "Accept: application/vnd.github.v3.full+json" \
+ -H "Content-Type: application/json" \
+ -H "${githubauth}" \
+ -d @"${restfile}" \
+ --silent --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/issues/${GITHUB_ISSUE}/comments" \
+ >/dev/null
+
+ retval=$?
+ rm "${restfile}"
return ${retval}
}
-
## @description Print out the finished details to the Github PR
## @audience private
## @stability evolving
@@ -38,14 +418,15 @@ function github_write_comment
## @param runresult
function github_finalreport
{
- local result=$1
- local i
- local commentfile=${PATCH_DIR}/commentfile
- local comment
+ declare result=$1
+ declare i
+ declare commentfile=${PATCH_DIR}/gitcommentfile.$$
+ declare comment
rm "${commentfile}" 2>/dev/null
- if [[ ${JENKINS} != "true" ]] ; then
+ if [[ ${JENKINS} != "true"
+ || -z ${GITHUB_ISSUE} ]] ; then
return 0
fi
@@ -54,15 +435,15 @@ function github_finalreport
add_footer_table "Console output" "${BUILD_URL}console"
if [[ ${result} == 0 ]]; then
- add_header_line ":confetti_ball: **+1 overall**"
+ echo ":confetti_ball: **+1 overall**" >> "${commentfile}"
else
- add_header_line ":broken_heart: **-1 overall**"
+ echo ":broken_heart: **-1 overall**" >> "${commentfile}"
fi
printf "\n\n\n\n" >> "${commentfile}"
i=0
- until [[ $i -eq ${#TP_HEADER[@]} ]]; do
+ until [[ ${i} -eq ${#TP_HEADER[@]} ]]; do
printf "%s\n\n" "${TP_HEADER[${i}]}" >> "${commentfile}"
((i=i+1))
done
@@ -74,7 +455,7 @@ function github_finalreport
} >> "${commentfile}"
i=0
- until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do
+ until [[ ${i} -eq ${#TP_VOTE_TABLE[@]} ]]; do
echo "${TP_VOTE_TABLE[${i}]}" >> "${commentfile}"
((i=i+1))
done
@@ -86,7 +467,7 @@ function github_finalreport
echo "|-------:|:------|"
} >> "${commentfile}"
i=0
- until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do
+ until [[ ${i} -eq ${#TP_TEST_TABLE[@]} ]]; do
echo "${TP_TEST_TABLE[${i}]}" >> "${commentfile}"
((i=i+1))
done
@@ -108,5 +489,5 @@ function github_finalreport
printf "\n\nThis message was automatically generated.\n\n" >> "${commentfile}"
- write_to_github "${commentfile}"
+ github_write_comment "${commentfile}"
}
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.d/jira.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/jira.sh b/dev-support/test-patch.d/jira.sh
index f95ca6f..ca9f2bd 100755
--- a/dev-support/test-patch.d/jira.sh
+++ b/dev-support/test-patch.d/jira.sh
@@ -14,26 +14,37 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-JIRACLI=${JIRA:-jira}
+# this bug system handles JIRA. Personalities
+# can override the following variables:
+
+# base JIRA URL
+JIRA_URL=${JIRA_URL:-"https://issues.apache.org/jira"}
+
+# Issue regex to help identify the project
+JIRA_ISSUE_RE=''
add_bugsystem jira
function jira_usage
{
echo "JIRA Options:"
- echo "--jira-cmd=<cmd> The 'jira' command to use (default 'jira')"
- echo "--jira-password=<pw> The password for the 'jira' command"
- echo "--jira-user=<user> The user for the 'jira' command"
+ echo "--jira-issue-re=<expr> Bash regular expression to use when trying to find a jira ref in the patch name (default: \'${JIRA_ISSUE_RE}\')"
+ echo "--jira-password=<pw> The password for the 'jira' command"
+ echo "--jira-base-url=<url> The URL of the JIRA server (default:'${JIRA_URL}')"
+ echo "--jira-user=<user> The user for the 'jira' command"
}
function jira_parse_args
{
- local i
+ declare i
for i in "$@"; do
case ${i} in
- --jira-cmd=*)
- JIRACLI=${i#*=}
+ --jira-base-url=*)
+ JIRA_URL=${i#*=}
+ ;;
+ --jira-issue-re=*)
+ JIRA_ISSUE_RE=${i#*=}
;;
--jira-password=*)
JIRA_PASSWD=${i#*=}
@@ -45,31 +56,248 @@ function jira_parse_args
done
}
+## @description provides issue determination based upon the URL and more.
+## @description WARNING: called from the github plugin!
+function jira_determine_issue
+{
+ declare input=$1
+ declare patchnamechunk
+ declare maybeissue
+
+ # shellcheck disable=SC2016
+ patchnamechunk=$(echo "${input}" | ${AWK} -F/ '{print $NF}')
+
+ maybeissue=$(echo "${patchnamechunk}" | cut -f1,2 -d-)
+
+ if [[ ${maybeissue} =~ ${JIRA_ISSUE_RE} ]]; then
+ ISSUE=${maybeissue}
+ JIRA_ISSUE=${maybeissue}
+ add_footer_table "JIRA Issue" "${ISSUE}"
+ return 0
+ fi
+
+ return 1
+}
+
+function jira_http_fetch
+{
+ declare input=$1
+ declare output=$2
+
+ if [[ -n "${JIRA_USER}"
+ && -n "${JIRA_PASSWD}" ]]; then
+ ${CURL} --silent --fail \
+ --user "${JIRA_USER}:${JIRA_PASSWD}" \
+ --output "${output}" \
+ --location \
+ "${JIRA_URL}/${input}"
+ else
+ ${CURL} --silent --fail \
+ --output "${output}" \
+ --location \
+ "${JIRA_URL}/${input}"
+ fi
+}
+
+function jira_locate_patch
+{
+ declare input=$1
+ declare fileloc=$2
+ declare relativeurl
+
+ yetus_debug "jira_locate_patch: trying ${JIRA_URL}/browse/${input}"
+
+ if [[ "${OFFLINE}" == true ]]; then
+ yetus_debug "jira_locate_patch: offline, skipping"
+ return 1
+ fi
+
+ jira_http_fetch "browse/${input}" "${PATCH_DIR}/jira"
+
+ if [[ $? != 0 ]]; then
+ yetus_debug "jira_locate_patch: not a JIRA."
+ return 1
+ fi
+
+ # if github is configured and we see what looks like a URL,
+ # send this to the github plugin to process.
+ if [[ -n "${GITHUB_BASE_URL}"
+ && $(${GREP} -c "${GITHUB_BASE_URL}"'[^ ]*patch' "${PATCH_DIR}/jira") != 0 ]]; then
+ echo "${input} appears to be a Github PR. Switching Modes."
+ github_jira_bridge "${fileloc}"
+ return $?
+ elif [[ $(${GREP} -c 'Patch Available' "${PATCH_DIR}/jira") == 0 ]]; then
+ if [[ ${JENKINS} == true ]]; then
+ yetus_error "ERROR: ${input} is not \"Patch Available\"."
+ cleanup_and_exit 1
+ else
+ yetus_error "WARNING: ${input} is not \"Patch Available\"."
+ fi
+ fi
+
+ #shellcheck disable=SC2016
+ relativeurl=$(${AWK} 'match($0,"/secure/attachment/[0-9]*/[^\"]*"){print substr($0,RSTART,RLENGTH)}' "${PATCH_DIR}/jira" |
+ ${GREP} -v -e 'htm[l]*$' | sort | tail -1 | ${SED} -e 's,[ ]*$,,g')
+ PATCHURL="${JIRA_URL}${relativeurl}"
+ if [[ ! ${PATCHURL} =~ \.patch$ ]]; then
+ guess_patch_file "${PATCH_DIR}/patch"
+ if [[ $? == 0 ]]; then
+ yetus_debug "The patch ${PATCHURL} was not named properly, but it looks like a patch file. Proceeding, but issue/branch matching might go awry."
+ add_vote_table 0 patch "The patch file was not named according to ${PROJECT_NAME}'s naming conventions. Please see ${HOW_TO_CONTRIBUTE} for instructions."
+ fi
+ fi
+ echo "${input} patch is being downloaded at $(date) from"
+ echo "${PATCHURL}"
+ add_footer_table "JIRA Patch URL" "${PATCHURL}"
+ jira_http_fetch "${relativeurl}" "${fileloc}"
+ if [[ $? != 0 ]];then
+ yetus_error "ERROR: ${input}/${PATCHURL} could not be downloaded."
+ cleanup_and_exit 1
+ fi
+ return 0
+}
+
+## @description Try to guess the branch being tested using a variety of heuristics
+## @audience private
+## @stability evolving
+## @replaceable no
+## @return 0 on success, with PATCH_BRANCH updated appropriately
+function jira_determine_branch
+{
+ declare patchnamechunk
+ declare total
+ declare count
+ declare hinttype
+
+ for hinttype in "${PATCHURL}" "${PATCH_OR_ISSUE}"; do
+ if [[ -z "${hinttype}" ]]; then
+ continue
+ fi
+
+ # If one of these matches the JIRA issue regex
+ # then we don't want it to trigger the branch
+ # detection since that's almost certainly not
+ # intended. In other words, if ISSUE-99 is the
+ # name of a branch, you want to test ISSUE-99
+ # against master, not ISSUE-99's branch
+ if [[ ${hinttype} =~ ${JIRA_ISSUE_RE} ]]; then
+ continue
+ fi
+
+ yetus_debug "Determine branch: starting with ${hinttype}"
+ patchnamechunk=$(echo "${hinttype}" \
+ | ${SED} -e 's,.*/\(.*\)$,\1,' \
+ -e 's,\.txt,.,' \
+ -e 's,.patch,.,g' \
+ -e 's,.diff,.,g' \
+ -e 's,\.\.,.,g' \
+ -e 's,\.$,,g' )
+
+ # ISSUE-branch-##
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1,2 -d-)
+ yetus_debug "Determine branch: ISSUE-branch-## = ${PATCH_BRANCH}"
+ if [[ -n "${PATCH_BRANCH}" ]]; then
+ verify_valid_branch "${PATCH_BRANCH}"
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ fi
+
+ # ISSUE-##[.##].branch
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d. )
+ count="${PATCH_BRANCH//[^.]}"
+ total=${#count}
+ ((total = total + 3 ))
+ until [[ ${total} -lt 2 ]]; do
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3-${total} -d.)
+ yetus_debug "Determine branch: ISSUE[.##].branch = ${PATCH_BRANCH}"
+ ((total=total-1))
+ if [[ -n "${PATCH_BRANCH}" ]]; then
+ verify_valid_branch "${PATCH_BRANCH}"
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ fi
+ done
+
+ # ISSUE.branch.##
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2- -d. )
+ count="${PATCH_BRANCH//[^.]}"
+ total=${#count}
+ ((total = total + 3 ))
+ until [[ ${total} -lt 2 ]]; do
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2-${total} -d.)
+ yetus_debug "Determine branch: ISSUE.branch[.##] = ${PATCH_BRANCH}"
+ ((total=total-1))
+ if [[ -n "${PATCH_BRANCH}" ]]; then
+ verify_valid_branch "${PATCH_BRANCH}"
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ fi
+ done
+
+ # ISSUE-branch.##
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1- -d. )
+ count="${PATCH_BRANCH//[^.]}"
+ total=${#count}
+ ((total = total + 1 ))
+ until [[ ${total} -eq 1 ]]; do
+ PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1-${total} -d. )
+ yetus_debug "Determine branch: ISSUE-branch[.##] = ${PATCH_BRANCH}"
+ ((total=total-1))
+ if [[ -n "${PATCH_BRANCH}" ]]; then
+ verify_valid_branch "${PATCH_BRANCH}"
+ if [[ $? == 0 ]]; then
+ return 0
+ fi
+ fi
+ done
+ done
+
+ return 1
+}
+
## @description Write the contents of a file to JIRA
## @params filename
## @stability stable
## @audience public
-## @returns ${JIRACLI} exit code
+## @returns exit code from posting to jira
function jira_write_comment
{
- local -r commentfile=${1}
- shift
-
- local retval=0
+ declare -r commentfile=${1}
+ declare retval=0
+ if [[ "${OFFLINE}" == true ]]; then
+ echo "JIRA Plugin: Running in offline, comment skipped."
+ return 0
+ fi
if [[ -n ${JIRA_PASSWD}
&& -n ${JIRA_USER} ]]; then
- # shellcheck disable=SC2086
- ${JIRACLI} --comment "$(cat ${commentfile})" \
- -s https://issues.apache.org/jira \
- -a addcomment -u ${JIRA_USER} \
- -p "${JIRA_PASSWD}" \
- --issue "${ISSUE}"
+
+ # RESTify the comment
+ {
+ echo "{\"body\":\""
+ ${SED} -e 's,\\,\\\\,g' \
+ -e 's,\",\\\",g' \
+ -e 's,$,\\r\\n,g' "${commentfile}" \
+ | tr -d '\n'
+ echo "\"}"
+ } > "${PATCH_DIR}/jiracomment.$$"
+
+ ${CURL} -X POST \
+ -H "Accept: application/json" \
+ -H "Content-Type: application/json" \
+ -u "${JIRA_USER}:${JIRA_PASSWD}" \
+ -d @"${PATCH_DIR}/jiracomment.$$" \
+ --silent --location \
+ "${JIRA_URL}/rest/api/2/issue/${JIRA_ISSUE}/comment" \
+ >/dev/null
retval=$?
- ${JIRACLI} -s https://issues.apache.org/jira \
- -a logout -u "${JIRA_USER}" \
- -p "${JIRA_PASSWD}"
+ rm "${PATCH_DIR}/jiracomment.$$"
+ else
+ echo "JIRA Plugin: no credentials provided to write a comment."
fi
return ${retval}
}
@@ -81,20 +309,25 @@ function jira_write_comment
## @param runresult
function jira_finalreport
{
- local result=$1
- local i
- local commentfile=${PATCH_DIR}/commentfile
- local comment
- local vote
- local ourstring
- local ela
- local subs
- local color
- local comment
+ declare result=$1
+ declare i
+ declare commentfile=${PATCH_DIR}/jiracommentfile
+ declare comment
+ declare vote
+ declare ourstring
+ declare ela
+ declare subs
+ declare color
+ declare comment
rm "${commentfile}" 2>/dev/null
- if [[ ${JENKINS} != "true" ]] ; then
+ if [[ ${JENKINS} == "false"
+ || ${OFFLINE} == true ]] ; then
+ return 0
+ fi
+
+ if [[ -z "${JIRA_ISSUE}" ]]; then
return 0
fi
@@ -103,12 +336,12 @@ function jira_finalreport
add_footer_table "Console output" "${BUILD_URL}console"
if [[ ${result} == 0 ]]; then
- add_header_line "| (/) *{color:green}+1 overall{color}* |"
+ echo "| (/) *{color:green}+1 overall{color}* |" >> "${commentfile}"
else
- add_header_line "| (x) *{color:red}-1 overall{color}* |"
+ echo "| (x) *{color:red}-1 overall{color}* |" >> "${commentfile}"
fi
- { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}"
+ echo "\\\\" >> "${commentfile}"
i=0
until [[ $i -eq ${#TP_HEADER[@]} ]]; do
@@ -116,7 +349,7 @@ function jira_finalreport
((i=i+1))
done
- { echo "\\\\" ; echo "\\\\"; } >> "${commentfile}"
+ echo "\\\\" >> "${commentfile}"
echo "|| Vote || Subsystem || Runtime || Comment ||" >> "${commentfile}"
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.d/shellcheck.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/shellcheck.sh b/dev-support/test-patch.d/shellcheck.sh
index 4d17768..0c198db 100755
--- a/dev-support/test-patch.d/shellcheck.sh
+++ b/dev-support/test-patch.d/shellcheck.sh
@@ -137,7 +137,10 @@ function shellcheck_postapply
fi
add_footer_table shellcheck "${msg}"
- calcdiffs "${PATCH_DIR}/branch-shellcheck-result.txt" "${PATCH_DIR}/patch-shellcheck-result.txt" > "${PATCH_DIR}/diff-patch-shellcheck.txt"
+ calcdiffs \
+ "${PATCH_DIR}/branch-shellcheck-result.txt" \
+ "${PATCH_DIR}/patch-shellcheck-result.txt" \
+ > "${PATCH_DIR}/diff-patch-shellcheck.txt"
# shellcheck disable=SC2016
diffPostpatch=$(wc -l "${PATCH_DIR}/diff-patch-shellcheck.txt" | ${AWK} '{print $1}')
@@ -151,6 +154,7 @@ function shellcheck_postapply
add_vote_table -1 shellcheck "The applied patch generated "\
"${diffPostpatch} new shellcheck issues (total was ${numPrepatch}, now ${numPostpatch})."
add_footer_table shellcheck "@@BASE@@/diff-patch-shellcheck.txt"
+ bugsystem_linecomments "shellcheck" "${PATCH_DIR}/diff-patch-shellcheck.txt"
return 1
fi
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.d/whitespace.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.d/whitespace.sh b/dev-support/test-patch.d/whitespace.sh
index 6fc033b..bab32dd 100755
--- a/dev-support/test-patch.d/whitespace.sh
+++ b/dev-support/test-patch.d/whitespace.sh
@@ -16,6 +16,26 @@
add_plugin whitespace
+
+function whitespace_linecomment_reporter
+{
+ local file=$1
+ shift
+ local comment=$*
+ local tmpfile="${PATCH_DIR}/wlr.$$.${RANDOM}"
+
+ while read -r line; do
+ {
+ #shellcheck disable=SC2086
+ printf "%s" "$(echo ${line} | cut -f1-2 -d:)"
+ echo "${comment}"
+ } >> "${tmpfile}"
+ done < "${file}"
+
+ bugsystem_linecomments "whitespace:" "${tmpfile}"
+ rm "${tmpfile}"
+}
+
function whitespace_postapply
{
local count
@@ -40,6 +60,8 @@ function whitespace_postapply
if [[ ${count} -gt 0 ]]; then
add_vote_table -1 whitespace "The patch has ${count}"\
" line(s) that end in whitespace. Use git apply --whitespace=fix."
+
+ whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-eol.txt" "end of line"
add_footer_table whitespace "@@BASE@@/whitespace-eol.txt"
((result=result+1))
fi
@@ -51,6 +73,7 @@ function whitespace_postapply
add_vote_table -1 whitespace "The patch has ${count}"\
" line(s) with tabs."
add_footer_table whitespace "@@BASE@@/whitespace-tabs.txt"
+ whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-tabs.txt" "tabs in line"
((result=result+1))
fi
http://git-wip-us.apache.org/repos/asf/yetus/blob/463891f3/dev-support/test-patch.sh
----------------------------------------------------------------------
diff --git a/dev-support/test-patch.sh b/dev-support/test-patch.sh
index a368f83..67dc3b4 100755
--- a/dev-support/test-patch.sh
+++ b/dev-support/test-patch.sh
@@ -98,23 +98,18 @@ function setup_defaults
REEXECED=false
RESETREPO=false
ISSUE=""
- ISSUE_RE='^(YETUS)-[0-9]+$'
TIMER=$(date +"%s")
- PATCHURL=""
OSTYPE=$(uname -s)
BUILDTOOL=maven
- BUGSYSTEM=jira
TESTFORMATS=""
JDK_TEST_LIST="javac javadoc unit"
- GITDIFFLINES="${PATCH_DIR}/gitdifflines.txt"
- GITDIFFCONTENT="${PATCH_DIR}/gitdiffcontent.txt"
# Solaris needs POSIX, not SVID
case ${OSTYPE} in
SunOS)
AWK=${AWK:-/usr/xpg4/bin/awk}
SED=${SED:-/usr/xpg4/bin/sed}
- WGET=${WGET:-wget}
+ CURL=${CURL:-curl}
GIT=${GIT:-git}
GREP=${GREP:-/usr/xpg4/bin/grep}
PATCH=${PATCH:-/usr/gnu/bin/patch}
@@ -124,7 +119,7 @@ function setup_defaults
*)
AWK=${AWK:-awk}
SED=${SED:-sed}
- WGET=${WGET:-wget}
+ CURL=${CURL:-curl}
GIT=${GIT:-git}
GREP=${GREP:-grep}
PATCH=${PATCH:-patch}
@@ -242,6 +237,7 @@ function offset_clock
## @param string
function add_header_line
{
+ # shellcheck disable=SC2034
TP_HEADER[${TP_HEADER_COUNTER}]="$*"
((TP_HEADER_COUNTER=TP_HEADER_COUNTER+1 ))
}
@@ -279,8 +275,10 @@ function add_vote_table
fi
if [[ -z ${value} ]]; then
+ # shellcheck disable=SC2034
TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| | ${subsystem} | | ${*:-} |"
else
+ # shellcheck disable=SC2034
TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| ${value} | ${subsystem} | ${calctime} | $* |"
fi
((TP_VOTE_COUNTER=TP_VOTE_COUNTER+1))
@@ -423,6 +421,7 @@ function finish_vote_table
echo "Total Elapsed time: ${calctime}"
echo ""
+ # shellcheck disable=SC2034
TP_VOTE_TABLE[${TP_VOTE_COUNTER}]="| | | ${calctime} | |"
((TP_VOTE_COUNTER=TP_VOTE_COUNTER+1 ))
}
@@ -440,6 +439,7 @@ function add_footer_table
local subsystem=$1
shift 1
+ # shellcheck disable=SC2034
TP_FOOTER_TABLE[${TP_FOOTER_COUNTER}]="| ${subsystem} | $* |"
((TP_FOOTER_COUNTER=TP_FOOTER_COUNTER+1 ))
}
@@ -455,6 +455,7 @@ function add_test_table
local failure=$1
shift 1
+ # shellcheck disable=SC2034
TP_TEST_TABLE[${TP_TEST_COUNTER}]="| ${failure} | $* |"
((TP_TEST_COUNTER=TP_TEST_COUNTER+1 ))
}
@@ -538,24 +539,21 @@ function find_java_home
return 0
}
-## @description Write the contents of a file to jenkins
+## @description Write the contents of a file to all of the bug systems
+## @description (so content should avoid special formatting)
## @params filename
## @stability stable
## @audience public
-## @returns ${JIRACLI} exit code
function write_comment
{
local -r commentfile=${1}
- shift
+ declare bug
- local retval=0
-
- if [[ ${OFFLINE} == false
- && ${JENKINS} == true ]]; then
- ${BUGSYSTEM}_write_comment "${commentfile}"
- retval=$?
- fi
- return ${retval}
+ for bug in ${BUGCOMMENTS}; do
+ if declare -f ${bug}_write_comment >/dev/null; then
+ "${bug}_write_comment" "${commentfile}"
+ fi
+ done
}
## @description Verify that the patch directory is still in working order
@@ -611,7 +609,7 @@ function compute_gitdiff
pushd "${BASEDIR}" >/dev/null
${GIT} add --all --intent-to-add
- while read line; do
+ while read -r line; do
if [[ ${line} =~ ^\+\+\+ ]]; then
file="./"$(echo "${line}" | cut -f2- -d/)
continue
@@ -651,16 +649,70 @@ function compute_gitdiff
fi
done < <("${GIT}" diff --unified=0 --no-color)
- if [[ ! -f ${GITDIFFLINES} ]]; then
+ if [[ ! -f "${GITDIFFLINES}" ]]; then
touch "${GITDIFFLINES}"
fi
- if [[ ! -f ${GITDIFFCONTENT} ]]; then
+
+ if [[ ! -f "${GITDIFFCONTENT}" ]]; then
touch "${GITDIFFCONTENT}"
fi
+ if [[ -s "${GITDIFFLINES}" ]]; then
+ compute_unidiff
+ else
+ touch "${GITUNIDIFFLINES}"
+ fi
+
popd >/dev/null
}
+## @description generate an index of unified diff lines vs. modified/added lines
+## @description ${GITDIFFLINES} must exist.
+## @audience private
+## @stability stable
+## @replaceable no
+function compute_unidiff
+{
+ declare fn
+ declare filen
+ declare tmpfile="${PATCH_DIR}/tmp.$$.${RANDOM}"
+
+ # now that we know what lines are where, we can deal
+ # with github's pain-in-the-butt API. It requires
+ # that the client provides the line number of the
+ # unified diff on a per file basis.
+
+ # First, build a per-file unified diff, pulling
+ # out the 'extra' lines, grabbing the adds with
+ # the line number in the diff file along the way,
+ # finally rewriting the line so that it is in
+ # './filename:diff line:content' format
+
+ for fn in ${CHANGED_FILES}; do
+ filen=${fn##./}
+
+ ${GIT} diff ${filen} \
+ | tail -n +6 \
+ | ${GREP} -n '^+' \
+ | ${GREP} -vE '^[0-9]*:\+\+\+' \
+ | ${SED} -e 's,^\([0-9]*:\)\+,\1,g' \
+ -e s,^,./${filen}:,g \
+ >> "${tmpfile}"
+ done
+
+ # at this point, tmpfile should be in the same format
+ # as gitdiffcontent, just with different line numbers.
+ # let's do a merge (using gitdifflines because it's easier)
+
+ # ./filename:real number:diff number
+ # shellcheck disable=SC2016
+ paste -d: "${GITDIFFLINES}" "${tmpfile}" \
+ | ${AWK} -F: '{print $1":"$2":"$5":"$6}' \
+ >> "${GITUNIDIFFLINES}"
+
+ rm "${tmpfile}"
+}
+
## @description Print the command to be executing to the screen. Then
## @description run the command, sending stdout and stderr to the given filename
## @description This will also ensure that any directories in ${BASEDIR} have
@@ -729,17 +781,16 @@ function testpatch_usage
echo "--basedir=<dir> The directory to apply the patch to (default current directory)"
echo "--branch=<ref> Forcibly set the branch"
echo "--branch-default=<ref> If the branch isn't forced and we don't detect one in the patch name, use this branch (default 'master')"
- #not quite working yet
- #echo "--bugsystem=<type> The bug system in use ('jira', the default, or 'github')"
echo "--build-native=<bool> If true, then build native components (default 'true')"
echo "--build-tool=<tool> Pick which build tool to focus around (maven, ant)"
+ echo "--bugcomments=<bug> Only write comments to the screen and this comma delimited list"
echo "--contrib-guide=<url> URL to point new users towards project conventions. (default: ${HOW_TO_CONTRIBUTE} )"
echo "--debug If set, then output some extra stuff to stderr"
echo "--dirty-workspace Allow the local git workspace to have uncommitted changes"
echo "--docker Spawn a docker container"
echo "--dockerfile=<file> Dockerfile fragment to use as the base"
- echo "--issue-re=<expr> Bash regular expression to use when trying to find a jira ref in the patch name (default: \'${ISSUE_RE}\')"
echo "--java-home=<path> Set JAVA_HOME (In Docker mode, this should be local to the image)"
+ echo "--linecomments=<bug> Only write line comments to this comma delimited list (defaults to bugcomments)"
echo "--multijdkdirs=<paths> Comma delimited lists of JDK paths to use for multi-JDK tests"
echo "--multijdktests=<list> Comma delimited tests to use when multijdkdirs is used. (default: javac,javadoc,unit)"
echo "--modulelist=<list> Specify additional modules to test (comma delimited)"
@@ -760,6 +811,7 @@ function testpatch_usage
echo "Shell binary overrides:"
echo "--ant-cmd=<cmd> The 'ant' command to use (default \${ANT_HOME}/bin/ant, or 'ant')"
echo "--awk-cmd=<cmd> The 'awk' command to use (default 'awk')"
+ echo "--curl-cmd=<cmd> The 'wget' command to use (default 'curl')"
echo "--diff-cmd=<cmd> The GNU-compatible 'diff' command to use (default 'diff')"
echo "--file-cmd=<cmd> The 'file' command to use (default 'file')"
echo "--git-cmd=<cmd> The 'git' command to use (default 'git')"
@@ -774,7 +826,6 @@ function testpatch_usage
echo "--build-url Set the build location web page"
echo "--eclipse-home=<path> Eclipse home directory (default ECLIPSE_HOME environment variable)"
echo "--mv-patch-dir Move the patch-dir into the basedir during cleanup."
- echo "--wget-cmd=<cmd> The 'wget' command to use (default 'wget')"
importplugins
@@ -815,8 +866,9 @@ function parse_args
--branch-default=*)
PATCH_BRANCH_DEFAULT=${i#*=}
;;
- --bugsystem=*)
- BUGSYSTEM=${i#*=}
+ --bugcomments=*)
+ BUGCOMMENTS=${i#*=}
+ BUGCOMMENTS=${BUGCOMMENTS//,/ }
;;
--build-native=*)
BUILD_NATIVE=${i#*=}
@@ -830,6 +882,9 @@ function parse_args
--contrib-guide=*)
HOW_TO_CONTRIBUTE=${i#*=}
;;
+ --curl-cmd=*)
+ CURL=${i#*=}
+ ;;
--debug)
TP_SHELL_SCRIPT_DEBUG=true
;;
@@ -864,9 +919,6 @@ function parse_args
testpatch_usage
exit 0
;;
- --issue-re=*)
- ISSUE_RE=${i#*=}
- ;;
--java-home=*)
JAVA_HOME=${i#*=}
;;
@@ -874,6 +926,10 @@ function parse_args
JENKINS=true
TEST_PARALLEL=${TEST_PARALLEL:-true}
;;
+ --linecomments=*)
+ BUGLINECOMMENTS=${i#*=}
+ BUGLINECOMMENTS=${BUGLINECOMMENTS//,/ }
+ ;;
--modulelist=*)
USER_MODULE_LIST=${i#*=}
USER_MODULE_LIST=${USER_MODULE_LIST//,/ }
@@ -954,9 +1010,6 @@ function parse_args
--tpreexectimer=*)
REEXECLAUNCHTIMER=${i#*=}
;;
- --wget-cmd=*)
- WGET=${i#*=}
- ;;
--*)
## PATCH_OR_ISSUE can't be a --. So this is probably
## a plugin thing.
@@ -1039,6 +1092,8 @@ function parse_args
GITDIFFLINES="${PATCH_DIR}/gitdifflines.txt"
GITDIFFCONTENT="${PATCH_DIR}/gitdiffcontent.txt"
+ GITUNIDIFFLINES="${PATCH_DIR}/gitdiffunilines.txt"
+
}
## @description Locate the build file for a given directory
@@ -1148,8 +1203,7 @@ function find_changed_modules
;;
*)
yetus_error "ERROR: Unsupported build tool."
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
;;
esac
@@ -1167,8 +1221,7 @@ function find_changed_modules
builddir=$(find_buildfile_dir ${buildfile} "${i}")
if [[ -z ${builddir} ]]; then
yetus_error "ERROR: ${buildfile} is not found. Make sure the target is a ${BUILDTOOL}-based project."
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
fi
builddirs="${builddirs} ${builddir}"
@@ -1188,6 +1241,8 @@ function find_changed_modules
buildmods="${buildmods} ${module}"
fi
done
+ else
+ buildmods=${CHANGED_UNFILTERED_MODULES}
fi
#shellcheck disable=SC2086,SC2034
@@ -1245,6 +1300,7 @@ function git_checkout
{
local currentbranch
local exemptdir
+ local status
big_console_header "Confirming git environment"
@@ -1342,8 +1398,6 @@ function git_checkout
determine_issue
GIT_REVISION=$(${GIT} rev-parse --verify --short HEAD)
- # shellcheck disable=SC2034
- VERSION=${GIT_REVISION}_${ISSUE}_PATCH-${patchNum}
if [[ "${ISSUE}" == 'Unknown' ]]; then
echo "Testing patch on ${PATCH_BRANCH}."
@@ -1402,9 +1456,8 @@ function verify_valid_branch
## @return 1 on failure, with PATCH_BRANCH updated to PATCH_BRANCH_DEFAULT
function determine_branch
{
- local patchnamechunk
- local total
- local count
+ declare bugs
+ declare retval=1
# something has already set this, so move on
if [[ -n ${PATCH_BRANCH} ]]; then
@@ -1427,83 +1480,19 @@ function determine_branch
return
fi
- for j in "${PATCHURL}" "${PATCH_OR_ISSUE}"; do
- if [[ -z "${j}" ]]; then
- continue
- fi
- yetus_debug "Determine branch: starting with ${j}"
- patchnamechunk=$(echo "${j}" \
- | ${SED} -e 's,.*/\(.*\)$,\1,' \
- -e 's,\.txt,.,' \
- -e 's,.patch,.,g' \
- -e 's,.diff,.,g' \
- -e 's,\.\.,.,g' \
- -e 's,\.$,,g' )
-
- # ISSUE-branch-##
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1,2 -d-)
- yetus_debug "Determine branch: ISSUE-branch-## = ${PATCH_BRANCH}"
- if [[ -n "${PATCH_BRANCH}" ]]; then
- verify_valid_branch "${PATCH_BRANCH}"
- if [[ $? == 0 ]]; then
- return
+ for bugs in ${BUGSYSTEMS}; do
+ if declare -f ${bugs}_determine_branch >/dev/null;then
+ "${bugs}_determine_branch"
+ retval=$?
+ if [[ ${retval} == 0 ]]; then
+ break
fi
fi
-
- # ISSUE-##[.##].branch
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d. )
- count="${PATCH_BRANCH//[^.]}"
- total=${#count}
- ((total = total + 3 ))
- until [[ ${total} -eq 2 ]]; do
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3-${total} -d.)
- yetus_debug "Determine branch: ISSUE[.##].branch = ${PATCH_BRANCH}"
- ((total=total-1))
- if [[ -n "${PATCH_BRANCH}" ]]; then
- verify_valid_branch "${PATCH_BRANCH}"
- if [[ $? == 0 ]]; then
- return
- fi
- fi
- done
-
- # ISSUE.branch.##
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2- -d. )
- count="${PATCH_BRANCH//[^.]}"
- total=${#count}
- ((total = total + 3 ))
- until [[ ${total} -eq 2 ]]; do
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f2-${total} -d.)
- yetus_debug "Determine branch: ISSUE.branch[.##] = ${PATCH_BRANCH}"
- ((total=total-1))
- if [[ -n "${PATCH_BRANCH}" ]]; then
- verify_valid_branch "${PATCH_BRANCH}"
- if [[ $? == 0 ]]; then
- return
- fi
- fi
- done
-
- # ISSUE-branch.##
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1- -d. )
- count="${PATCH_BRANCH//[^.]}"
- total=${#count}
- ((total = total + 1 ))
- until [[ ${total} -eq 1 ]]; do
- PATCH_BRANCH=$(echo "${patchnamechunk}" | cut -f3- -d- | cut -f1-${total} -d. )
- yetus_debug "Determine branch: ISSUE-branch[.##] = ${PATCH_BRANCH}"
- ((total=total-1))
- if [[ -n "${PATCH_BRANCH}" ]]; then
- verify_valid_branch "${PATCH_BRANCH}"
- if [[ $? == 0 ]]; then
- return
- fi
- fi
- done
-
done
- PATCH_BRANCH="${PATCH_BRANCH_DEFAULT}"
+ if [[ ${retval} != 0 ]]; then
+ PATCH_BRANCH="${PATCH_BRANCH_DEFAULT}"
+ fi
popd >/dev/null
}
@@ -1515,28 +1504,19 @@ function determine_branch
## @return 1 on failure, with ISSUE updated to "Unknown"
function determine_issue
{
- local patchnamechunk
- local maybeissue
+ local bugsys
yetus_debug "Determine issue"
- # we can shortcut jenkins
- if [[ ${JENKINS} == true ]]; then
- ISSUE=${PATCH_OR_ISSUE}
- return 0
- fi
-
- # shellcheck disable=SC2016
- patchnamechunk=$(echo "${PATCH_OR_ISSUE}" | ${AWK} -F/ '{print $NF}')
-
- maybeissue=$(echo "${patchnamechunk}" | cut -f1,2 -d-)
-
- if [[ ${maybeissue} =~ ${ISSUE_RE} ]]; then
- ISSUE=${maybeissue}
- return 0
- fi
-
- ISSUE="Unknown"
+ for bugsys in ${BUGSYSTEMS}; do
+ if declare -f ${bugsys}_determine_issue >/dev/null; then
+ "${bugsys}_determine_issue" "${PATCH_OR_ISSUE}"
+ if [[ $? == 0 ]]; then
+ yetus_debug "${bugsys} says ${ISSUE}"
+ return 0
+ fi
+ fi
+ done
return 1
}
@@ -1585,6 +1565,7 @@ function verify_needed_test
function determine_needed_tests
{
local i
+ local plugin
for i in ${CHANGED_FILES}; do
yetus_debug "Determining needed tests for ${i}"
@@ -1609,100 +1590,51 @@ function determine_needed_tests
## @return 1 on failure, may exit
function locate_patch
{
- local notSureIfPatch=false
+ local bugsys
+ local patchfile=""
+ local gotit=false
+
yetus_debug "locate patch"
+ # it's a locally provided file
if [[ -f ${PATCH_OR_ISSUE} ]]; then
- PATCH_FILE="${PATCH_OR_ISSUE}"
+ patchfile="${PATCH_OR_ISSUE}"
else
- if [[ ${PATCH_OR_ISSUE} =~ ^http ]]; then
- echo "Patch is being downloaded at $(date) from"
- PATCHURL="${PATCH_OR_ISSUE}"
- else
- ${WGET} -q -O "${PATCH_DIR}/jira" "http://issues.apache.org/jira/browse/${PATCH_OR_ISSUE}"
-
- case $? in
- 0)
- ;;
- 2)
- yetus_error "ERROR: .wgetrc/.netrc parsing error."
- cleanup_and_exit 1
- ;;
- 3)
- yetus_error "ERROR: File IO error."
- cleanup_and_exit 1
- ;;
- 4)
- yetus_error "ERROR: URL ${PATCH_OR_ISSUE} is unreachable."
- cleanup_and_exit 1
- ;;
- *)
- # we want to try and do as much as we can in docker mode,
- # but if the patch was passed as a file, then we may not
- # be able to continue.
- if [[ ${REEXECED} == true
- && -f "${PATCH_DIR}/patch" ]]; then
- PATCH_FILE="${PATCH_DIR}/patch"
- else
- yetus_error "ERROR: Unable to fetch ${PATCH_OR_ISSUE}."
- cleanup_and_exit 1
- fi
- ;;
- esac
-
- if [[ -z "${PATCH_FILE}" ]]; then
- if [[ $(${GREP} -c 'Patch Available' "${PATCH_DIR}/jira") == 0 ]] ; then
- if [[ ${JENKINS} == true ]]; then
- yetus_error "ERROR: ${PATCH_OR_ISSUE} is not \"Patch Available\"."
- cleanup_and_exit 1
- else
- yetus_error "WARNING: ${PATCH_OR_ISSUE} is not \"Patch Available\"."
+ # run through the bug systems. maybe they know?
+ for bugsys in ${BUGSYSTEMS}; do
+ if declare -f ${bugsys}_locate_patch >/dev/null 2>&1; then
+ "${bugsys}_locate_patch" "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch"
+ if [[ $? == 0 ]]; then
+ guess_patch_file "${PATCH_DIR}/patch"
+ if [[ $? == 0 ]]; then
+ gotit=true
+ break;
fi
fi
-
- #shellcheck disable=SC2016
- relativePatchURL=$(${AWK} 'match($0,"\"/jira/secure/attachment/[0-9]*/[^\"]*"){print substr($0,RSTART+1,RLENGTH-1)}' "${PATCH_DIR}/jira" |
- ${GREP} -v -e 'htm[l]*$' | sort | tail -1)
- PATCHURL="http://issues.apache.org${relativePatchURL}"
- if [[ ! ${PATCHURL} =~ \.patch$ ]]; then
- notSureIfPatch=true
- fi
- #shellcheck disable=SC2016
- patchNum=$(echo "${PATCHURL}" | ${AWK} 'match($0,"[0-9]*/"){print substr($0,RSTART,RLENGTH-1)}')
- echo "${ISSUE} patch is being downloaded at $(date) from"
fi
- fi
- if [[ -z "${PATCH_FILE}" ]]; then
- echo "${PATCHURL}"
- add_footer_table "Patch URL" "${PATCHURL}"
- ${WGET} -q -O "${PATCH_DIR}/patch" "${PATCHURL}"
- if [[ $? != 0 ]];then
- yetus_error "ERROR: ${PATCH_OR_ISSUE} could not be downloaded."
- cleanup_and_exit 1
- fi
- PATCH_FILE="${PATCH_DIR}/patch"
+ done
+
+ # ok, none of the bug systems know. let's see how smart we are
+ if [[ ${gotit} == false ]]; then
+ generic_locate_patch "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch"
fi
fi
- if [[ ! -f "${PATCH_DIR}/patch" ]]; then
- cp "${PATCH_FILE}" "${PATCH_DIR}/patch"
+ if [[ ! -f "${PATCH_DIR}/patch"
+ && -f "${patchfile}" ]]; then
+ cp "${patchfile}" "${PATCH_DIR}/patch"
if [[ $? == 0 ]] ; then
- echo "Patch file ${PATCH_FILE} copied to ${PATCH_DIR}"
+ echo "Patch file ${patchfile} copied to ${PATCH_DIR}"
else
- yetus_error "ERROR: Could not copy ${PATCH_FILE} to ${PATCH_DIR}"
+ yetus_error "ERROR: Could not copy ${patchfile} to ${PATCH_DIR}"
cleanup_and_exit 1
fi
fi
- if [[ ${notSureIfPatch} == "true" ]]; then
- guess_patch_file "${PATCH_DIR}/patch"
- if [[ $? != 0 ]]; then
- yetus_error "ERROR: ${PATCHURL} is not a patch file."
- cleanup_and_exit 1
- else
- yetus_debug "The patch ${PATCHURL} was not named properly, but it looks like a patch file. proceeding, but issue/branch matching might go awry."
- add_vote_table 0 patch "The patch file was not named according to ${PROJECT_NAME}'s naming conventions. Please see ${HOW_TO_CONTRIBUTE} for instructions."
- fi
+ guess_patch_file "${PATCH_DIR}/patch"
+ if [[ $? != 0 ]]; then
+ yetus_error "ERROR: Unsure how to process ${PATCH_OR_ISSUE}."
+ cleanup_and_exit 1
fi
}
@@ -1717,6 +1649,10 @@ function guess_patch_file
local patch=$1
local fileOutput
+ if [[ ! -f ${patch} ]]; then
+ return 1
+ fi
+
yetus_debug "Trying to guess is ${patch} is a patch file."
fileOutput=$("${FILE}" "${patch}")
if [[ $fileOutput =~ \ diff\ ]]; then
@@ -1769,8 +1705,7 @@ function apply_patch_file
echo "PATCH APPLICATION FAILED"
((RESULT = RESULT + 1))
add_vote_table -1 patch "The patch command could not apply the patch."
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
fi
return 0
@@ -3064,134 +2999,55 @@ function check_unittests
return 0
}
-## @description Print out the finished details on the console
-## @audience private
+## @description Write comments onto bug systems that have code review support.
+## @description File should be in the form of "file:line:comment"
+## @audience public
## @stability evolving
## @replaceable no
-## @param runresult
-## @return 0 on success
-## @return 1 on failure
-function output_to_console
-{
- local result=$1
- shift
- local i=0
- local ourstring
- local vote
- local subs
- local ela
- local comment
- local commentfile1="${PATCH_DIR}/comment.1"
- local commentfile2="${PATCH_DIR}/comment.2"
- local normaltop
- local line
- local seccoladj=0
- local spcfx=${PATCH_DIR}/spcl.txt
-
- if [[ ${result} == 0 ]]; then
- if [[ ${JENKINS} == false ]]; then
- {
- printf "IF9fX19fX19fX18gCjwgU3VjY2VzcyEgPgogLS0tLS0tLS0tLSAKIFwgICAg";
- printf "IC9cICBfX18gIC9cCiAgXCAgIC8vIFwvICAgXC8gXFwKICAgICAoKCAgICBP";
- printf "IE8gICAgKSkKICAgICAgXFwgLyAgICAgXCAvLwogICAgICAgXC8gIHwgfCAg";
- printf "XC8gCiAgICAgICAgfCAgfCB8ICB8ICAKICAgICAgICB8ICB8IHwgIHwgIAog";
- printf "ICAgICAgIHwgICBvICAgfCAgCiAgICAgICAgfCB8ICAgfCB8ICAKICAgICAg";
- printf "ICB8bXwgICB8bXwgIAo"
- } > "${spcfx}"
- fi
- printf "\n\n+1 overall\n\n"
- else
- if [[ ${JENKINS} == false ]]; then
- {
- printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf";
- printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf";
- printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98";
- printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg";
- printf "ICAgICAgICAgICAgICAgICAK"
- } > "${spcfx}"
- fi
- printf "\n\n-1 overall\n\n"
- fi
-
- if [[ -f ${spcfx} ]]; then
- if which base64 >/dev/null 2>&1; then
- base64 --decode "${spcfx}" 2>/dev/null
- elif which openssl >/dev/null 2>&1; then
- openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null
- fi
- echo
- echo
- rm "${spcfx}"
- fi
-
- seccoladj=$(findlargest 2 "${TP_VOTE_TABLE[@]}")
- if [[ ${seccoladj} -lt 10 ]]; then
- seccoladj=10
+## @param filename
+function bugsystem_linecomments
+{
+ declare title=$1
+ declare fn=$2
+ declare line
+ declare bugs
+ declare realline
+ declare text
+ declare idxline
+ declare uniline
+
+ if [[ ! -f "${GITUNIDIFFLINES}" ]]; then
+ return
fi
- seccoladj=$((seccoladj + 2 ))
- i=0
- until [[ $i -eq ${#TP_HEADER[@]} ]]; do
- printf "%s\n" "${TP_HEADER[${i}]}"
- ((i=i+1))
- done
-
- printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment"
- echo "============================================================================"
- i=0
- until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do
- ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ')
- vote=$(echo "${ourstring}" | cut -f2 -d\|)
- subs=$(echo "${ourstring}" | cut -f3 -d\|)
- ela=$(echo "${ourstring}" | cut -f4 -d\|)
- comment=$(echo "${ourstring}" | cut -f5 -d\|)
-
- echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}"
- normaltop=$(head -1 "${commentfile1}")
- ${SED} -e '1d' "${commentfile1}" > "${commentfile2}"
-
- printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \
- "${subs}" "${ela}" "${normaltop}"
- while read line; do
- printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}"
- done < "${commentfile2}"
-
- ((i=i+1))
- rm "${commentfile2}" "${commentfile1}" 2>/dev/null
- done
+ while read -r line;do
+ file=$(echo "${line}" | cut -f1 -d:)
+ realline=$(echo "${line}" | cut -f2 -d:)
+ text=$(echo "${line}" | cut -f3- -d:)
+ idxline="${file}:${realline}:"
+ uniline=$(${GREP} "${idxline}" "${GITUNIDIFFLINES}" | cut -f3 -d: )
- if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then
- seccoladj=$(findlargest 1 "${TP_TEST_TABLE[@]}")
- printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason"
- i=0
- until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do
- ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ')
- vote=$(echo "${ourstring}" | cut -f2 -d\|)
- subs=$(echo "${ourstring}" | cut -f3 -d\|)
- printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}"
- ((i=i+1))
+ for bugs in ${BUGLINECOMMENTS}; do
+ if declare -f ${bugs}_linecomments >/dev/null;then
+ "${bugs}_linecomments" "${title}" "${file}" "${realline}" "${uniline}" "${text}"
+ fi
done
- fi
-
- printf "\n\n|| Subsystem || Report/Notes ||\n"
- echo "============================================================================"
- i=0
-
- until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do
- comment=$(echo "${TP_FOOTER_TABLE[${i}]}" |
- ${SED} -e "s,@@BASE@@,${PATCH_DIR},g")
- printf "%s\n" "${comment}"
- ((i=i+1))
- done
+ done < "${fn}"
}
## @description Write the final output to the selected bug system
## @audience private
## @stability evolving
## @replaceable no
-function output_to_bugsystem
+function bugsystem_finalreport
{
- "${BUGSYSTEM}_finalreport" "${@}"
+ declare bugs
+
+ for bugs in ${BUGCOMMENTS}; do
+ if declare -f ${bugs}_finalreport >/dev/null;then
+ "${bugs}_finalreport" "${@}"
+ fi
+ done
}
## @description Clean the filesystem as appropriate and then exit
@@ -3238,8 +3094,7 @@ function postcheckout
(( RESULT = RESULT + $? ))
if [[ ${RESULT} != 0 ]] ; then
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
fi
done
@@ -3255,8 +3110,7 @@ function postcheckout
(( RESULT = RESULT + $? ))
if [[ ${RESULT} != 0 ]] ; then
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
fi
fi
@@ -3312,8 +3166,7 @@ function postapply
check_patch_javac
retval=$?
if [[ ${retval} -gt 1 ]] ; then
- output_to_console 1
- output_to_bugsystem 1
+ bugsystem_finalreport 1
cleanup_and_exit 1
fi
@@ -3457,6 +3310,13 @@ function parse_args_plugins
(( RESULT = RESULT + $? ))
fi
done
+
+ BUGCOMMENTS=${BUGCOMMENTS:-${BUGSYSTEMS}}
+ if [[ ! ${BUGCOMMENTS} =~ console ]]; then
+ BUGCOMMENTS="${BUGCOMMENTS} console"
+ fi
+
+ BUGLINECOMMENTS=${BUGLINECOMMENTS:-${BUGCOMMENTS}}
}
## @description Register test-patch.d plugins
@@ -3588,6 +3448,5 @@ finish_vote_table
finish_footer_table
-output_to_console ${RESULT}
-output_to_bugsystem ${RESULT}
+bugsystem_finalreport ${RESULT}
cleanup_and_exit ${RESULT}