You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@yetus.apache.org by aw...@apache.org on 2020/11/02 14:28:28 UTC
[yetus] branch YETUS-1034-release updated (879f861 -> 6539d23)
This is an automated email from the ASF dual-hosted git repository.
aw pushed a change to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git.
from 879f861 YETUS-1061. remove github user and password support (addendum) (#187)
new 1d5d350 YETUS-1066. fix spelling mistakes (#186)
new 7977c8f YETUS-1064. buf does not generate annotations (#185)
new 97b10e4 YETUS-1067. mvn site fails on empty release (#189)
new 38c78a7 YETUS-1059. Github Status Recovery Tool (#184)
new 6539d23 YETUS-1063. Enhance documentaton for docker-cleanup and docker in gen… (#188)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.yetus/blanks-eol.txt => .codespellrc | 3 +-
Jenkinsfile | 30 ++-
asf-site-src/pom.xml | 1 +
asf-site-src/source/contribute.html.md | 2 +-
asf-site-src/source/contribute/releases.html.md | 2 +-
.../source/documentation/in-progress.html.md | 4 +-
.../in-progress/precommit/buildtools.html.md | 2 +-
.../in-progress/precommit/docker-cleanup.html.md | 14 +-
.../in-progress/precommit/docker.html.md | 66 ++++-
.../precommit/github-status-recovery.html.md | 50 ++++
.../in-progress/precommit/index.html.md | 1 +
.../in-progress/precommit/robots/index.html.md | 4 +-
.../in-progress/precommit/robots/jenkins.html.md | 23 +-
precommit/src/main/shell/core.d/00-yetuslib.sh | 2 +-
precommit/src/main/shell/core.d/01-common.sh | 2 +-
precommit/src/main/shell/core.d/docker.sh | 4 +-
precommit/src/main/shell/core.d/linecomments.sh | 132 ++++++++++
...docker-cleanup.sh => github-status-recovery.sh} | 86 ++-----
precommit/src/main/shell/robots.d/jenkins.sh | 2 +-
.../src/main/shell/test-patch-docker/Dockerfile | 2 +-
precommit/src/main/shell/test-patch.d/buf.sh | 27 ++-
precommit/src/main/shell/test-patch.d/github.sh | 265 ++++++++++++++++++++-
precommit/src/main/shell/test-patch.sh | 122 +---------
releasedocmaker/pom.xml | 1 +
.../src/main/python/releasedocmaker/__init__.py | 2 +-
.../src/main/python/releasedocmaker/utils.py | 2 +-
shelldocs/pom.xml | 6 +-
start-build-env.sh | 2 +-
28 files changed, 630 insertions(+), 229 deletions(-)
copy .yetus/blanks-eol.txt => .codespellrc (87%)
create mode 100644 asf-site-src/source/documentation/in-progress/precommit/github-status-recovery.html.md
create mode 100755 precommit/src/main/shell/core.d/linecomments.sh
copy precommit/src/main/shell/{docker-cleanup.sh => github-status-recovery.sh} (78%)
[yetus] 05/05: YETUS-1063. Enhance documentaton for docker-cleanup and docker in gen… (#188)
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 6539d2378945cb2468312adfaa6c28a0206f6bd5
Author: Allen Wittenauer <aw...@apache.org>
AuthorDate: Mon Nov 2 06:18:16 2020 -0800
YETUS-1063. Enhance documentaton for docker-cleanup and docker in gen… (#188)
---
.../in-progress/precommit/docker-cleanup.html.md | 14 ++++-
.../in-progress/precommit/docker.html.md | 64 ++++++++++++++++++++++
.../in-progress/precommit/robots/index.html.md | 4 +-
3 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
index c16af5c..59a5416 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
@@ -21,15 +21,25 @@
<!-- MarkdownTOC levels="1,2" autolink="true" indent=" " bullets="*" bracket="round" -->
+* [Description](#description)
* [Usage](#usage)
+ * [Default Mode](#default-mode)
+ * [Sentinel Mode](#sentinel-mode)
<!-- /MarkdownTOC -->
+# Description
+
`docker-cleanup` is a command to perform precommit's Docker cleanup functionality outside of patch and build testing.
+It is designed to be run as a regularly automated job on CI systems that do not have their own Docker cleanup
+facilities. It is built from the perspective that nothing on the system is permanent. It supports many of the same
+options as `test-patch`'s [docker](../docker)-mode. Please see that page for more details.
# Usage
-Executing `docker-cleanup` will perform the same Docker image as the `--robot` option.
+## Default Mode
+
+Executing `docker-cleanup` with no additional options will perform the same work as the `--robot` option.
```bash
$ docker-cleanup
@@ -45,6 +55,8 @@ Deleted: sha256:1a96c79a0a9ab538c6c7765dc908eca3e689270b778d6ae2add558e89792a7d8
```
+## Sentinel Mode
+
`docker-cleanup` also supports the `--sentinel` mode to kill and remove stale running containers:
```bash
diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
index 592181c..30b454d 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
@@ -28,9 +28,15 @@
* [Pulling a Docker tag](#pulling-a-docker-tag)
* [Using a cache](#using-a-cache)
* [Platforms](#platforms)
+ * [Container Directory](#container-directory)
* [Resource Controls](#resource-controls)
* [Privileged Mode](#privileged-mode)
* [Docker in Docker](#docker-in-docker)
+* [Cleaning the Docker Environment](#cleaning-the-docker-environment)
+ * [Images](#images)
+ * [Containers](#containers)
+ * [Dry Running Cleaning](#dry-running-cleaning)
+ * [Standalone Facility](#standalone-facility)
<!-- /MarkdownTOC -->
@@ -68,6 +74,12 @@ With the `--docker-cache-from` parameter, other images may be utilized to provid
When either building or pull an image, test-patch supports the `--docker-platform` flag to pass in the Docker `--platform` flag. This allows you full control over what kind of image the software either creates or fetches.
+## Container Directory
+
+By default, precommit will use `/precommit` as the directory where it will store any necessary components that are
+not provided by other flags in system (such as `--basedir` or `--patch-dir`). If that directory conflicts with some other
+need, then the `--docker-work-dir` option may be provided to set a different path.
+
# Resource Controls
Docker's `--memory` flag is supported via the `--dockermemlimit` option. This enables the container's memory size to be limited. This may be important to set to prevent things like broken unit tests bringing down the entire build server. See [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/) for more details. Apache Yetus also sets the `--oom-score-adj` to 500 in order to offer itself as the first processes to be killed if memory is low.
@@ -83,3 +95,55 @@ In some cases, the work being performed inside the Docker container requires ext
With the usage of the `--dockerind` flag, test-patch will mount the `/var/run/docker.sock` UNIX socket into the container to enable Docker-in-Docker mode. Additionally, the `--docker-socket` option will let one set the socket to mount in situations where that isn't the location of the socket, such as a dockerd proxy providing authentication.
NOTE: Using --dockerind requires the availability of the `stat` command that supports either -c '%g' (GNU form) or -f '%g' (BSD form).
+
+# Cleaning the Docker Environment
+
+With the growing use of Docker for CI purposes, so is the growing need to help manage the Docker environment. Stale, large
+caches and stuck containers are a common problem when tests go haywire. In order to help combat that, Apache Yetus includes
+facilities as part of precommit to help keep these things under control. By default, none of these cleaning actions occur.
+However if the system is in [Robot](../robots)-mode or `--sentinel` is used, carefully controlled cleaning will happen.
+
+ NOTE: Docker volumes are not managed by Apache Yetus.
+
+## Images
+
+For container images, there are three modes. Each mode increases the amount of coverage that Apache Yetus will use to
+remove images.
+
+| Mode | Images Types | Time Frame |
+|:----:|:------------:|:----------:|
+| default | all | NA |
+| `--robot` | dangling, label=org.apache.yetus | 1 week |
+| `--sentinel` | all | 1 week |
+
+Before it begins, Apache Yetus will show the contents of `docker images` so that there is a history of what was in
+the cache. This also helps determine what was deleted later. Next, it finds images that fall within the time frame of
+the given mode. Then it will untag those images and followed by trying to remove them by sha. If an
+image is in use, an error will be reported but ignored by Apache Yetus. It is an information message to help give the user
+an idea of what is happening under the hood.
+
+## Containers
+
+For containers there are two modes:
+
+| Mode | Images Types | Time Frame |
+|:----:|:------------:|:----------:|
+| default | all | NA |
+| `--sentinel` | all | 24 hours |
+
+Under just plain `--robot`, containers are left alone. Under `--sentinel`, containers (regardless of state) are
+killed after 24 hours.
+
+Just as with images, Apache Yetus will report all the running containers on the system. Then it will, if
+necessary, send a `docker kill` followed by a `docker rm` remove any containers that are over the runtime
+limit.
+
+## Dry Running Cleaning
+
+To see what the system would have done, the `--dockerdelrep` option is provided to report, but not act, on deletions.
+
+## Standalone Facility
+
+By popular demand, Apache Yetus provides the [docker-cleanup](../docker-cleanup) program to perform these functions
+outside of running jobs. This program is useful for CI systems that do not regularly run Apache Yetus or for
+simply consolidating the maintenance into one location.
diff --git a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
index 6f63bca..41431ae 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/robots/index.html.md
@@ -76,7 +76,9 @@ If `--robot` is specified without an instance, a random number is generated and
# Sentinel Mode
-If stuck Docker containers are a problem, a more aggressive robot may be enabled with the `--sentinel` option. This option enables killing containers that have been running for over 24 hours as well.
+If stuck Docker containers are a problem, a more aggressive robot may be enabled with the `--sentinel` option. This option
+enables killing containers that have been running for over 24 hours as well. See the documentation on [docker](../docker)
+for more information.
# Continuous Improvement
[yetus] 02/05: YETUS-1064. buf does not generate annotations (#185)
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 7977c8f11705c0e3adf8b5e6b5a7c3f2535e9774
Author: Allen Wittenauer <aw...@apache.org>
AuthorDate: Mon Nov 2 06:17:33 2020 -0800
YETUS-1064. buf does not generate annotations (#185)
---
precommit/src/main/shell/test-patch.d/buf.sh | 27 ++++++++++++++-------------
1 file changed, 14 insertions(+), 13 deletions(-)
diff --git a/precommit/src/main/shell/test-patch.d/buf.sh b/precommit/src/main/shell/test-patch.d/buf.sh
index b3b2aa9..23682fb 100755
--- a/precommit/src/main/shell/test-patch.d/buf.sh
+++ b/precommit/src/main/shell/test-patch.d/buf.sh
@@ -52,6 +52,12 @@ function buf_parse_args
;;
esac
done
+
+ if [[ -n "${BUF_BASEDIR}" ]]; then
+ # make sure this is relative and strip any ending /
+ BUF_BASEDIR=$(yetus_abs "${BUF_BASEDIR}")
+ BUF_BASEDIR=$(yetus_relative_dir "${BASEDIR}" "${BUF_BASEDIR}")
+ fi
}
function buf_filefilter
@@ -131,19 +137,17 @@ function bufcompat_executor
bufargs+=(--timeout "${BUF_TIMEOUT}")
fi
- if [[ -n "${BUF_BASEDIR}" ]]; then
- pushd "${BASEDIR}/${BUF_BASEDIR}" >/dev/null || return 1
- else
- pushd "${BASEDIR}" >/dev/null || return 1
- fi
+ pushd "${BASEDIR}/${BUF_BASEDIR}" >/dev/null || return 1
if [[ "${repostatus}" == "branch" ]]; then
"${BUF}" image build "${bufargs[@]}" -o "${PATCH_DIR}/buf-image.bin" 2>> "${PATCH_DIR}/${bufStderr}"
elif [[ -f "${PATCH_DIR}/buf-image.bin" ]]; then
"${BUF}" check breaking "${bufargs[@]}" \
--against-input "${PATCH_DIR}/buf-image.bin" \
+ 2>> "${PATCH_DIR}/${bufStderr}" \
+ | "${AWK}" "{print \"${BUF_BASEDIR}/\"\$0}" \
> "${PATCH_DIR}/${repostatus}-bufcompat-result.txt" \
- 2>> "${PATCH_DIR}/${bufStderr}"
+
fi
popd >/dev/null || return 1
@@ -259,14 +263,11 @@ function buflint_executor
bufargs+=(--timeout "${BUF_TIMEOUT}")
fi
- if [[ -n "${BUF_BASEDIR}" ]]; then
- pushd "${BASEDIR}/${BUF_BASEDIR}" >/dev/null || return 1
- else
- pushd "${BASEDIR}" >/dev/null || return 1
- fi
+ pushd "${BASEDIR}/${BUF_BASEDIR}" >/dev/null || return 1
- "${BUF}" check lint "${bufargs[@]}" 2>> "${PATCH_DIR}/${bufStderr}" | \
- "${args[@]}" > "${PATCH_DIR}/${repostatus}-buflint-result.txt"
+ "${BUF}" check lint "${bufargs[@]}" 2>> "${PATCH_DIR}/${bufStderr}" \
+ | "${AWK}" "{print \"${BUF_BASEDIR}/\"\$0}" \
+ | "${args[@]}" > "${PATCH_DIR}/${repostatus}-buflint-result.txt"
popd >/dev/null || return 1
[yetus] 01/05: YETUS-1066. fix spelling mistakes (#186)
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 1d5d3507865fe2147fe28b26cbc80096c808d4b3
Author: Allen Wittenauer <aw...@apache.org>
AuthorDate: Sat Oct 31 14:52:53 2020 -0700
YETUS-1066. fix spelling mistakes (#186)
---
.codespellrc | 17 +++++++++++++++++
Jenkinsfile | 2 +-
asf-site-src/source/contribute.html.md | 2 +-
asf-site-src/source/contribute/releases.html.md | 2 +-
asf-site-src/source/documentation/in-progress.html.md | 4 ++--
.../in-progress/precommit/buildtools.html.md | 2 +-
.../in-progress/precommit/docker-cleanup.html.md | 2 +-
.../documentation/in-progress/precommit/docker.html.md | 2 +-
precommit/src/main/shell/core.d/00-yetuslib.sh | 2 +-
precommit/src/main/shell/core.d/01-common.sh | 2 +-
precommit/src/main/shell/core.d/docker.sh | 4 ++--
precommit/src/main/shell/robots.d/jenkins.sh | 2 +-
precommit/src/main/shell/test-patch-docker/Dockerfile | 2 +-
precommit/src/main/shell/test-patch.d/github.sh | 6 +++---
precommit/src/main/shell/test-patch.sh | 8 ++++----
.../src/main/python/releasedocmaker/__init__.py | 2 +-
.../src/main/python/releasedocmaker/utils.py | 2 +-
start-build-env.sh | 2 +-
18 files changed, 41 insertions(+), 24 deletions(-)
diff --git a/.codespellrc b/.codespellrc
new file mode 100644
index 0000000..f34e714
--- /dev/null
+++ b/.codespellrc
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[codespell]
+skip = ./*.css.*,./*.js,./.git/*,./.asf.yaml,./asf-site-src/config.rb,./asf-site-src/Gemfile.lock
\ No newline at end of file
diff --git a/Jenkinsfile b/Jenkinsfile
index 5bbcfd9..0840c93 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -64,7 +64,7 @@ pipeline {
USE_DOCKER_FLAG=${USE_DOCKER_FLAG:-true}
# clean and make a new directory for our output artifacts, temporary
- # storage, etc just incase the workspace directory
+ # storage, etc just in case the workspace directory
# delete in post is removed
if [[ -d "${WORKSPACE}/${YETUS_RELATIVE_PATCHDIR}" ]]; then
rm -rf "${WORKSPACE}/${YETUS_RELATIVE_PATCHDIR}"
diff --git a/asf-site-src/source/contribute.html.md b/asf-site-src/source/contribute.html.md
index b8d9801..ab5ec7f 100644
--- a/asf-site-src/source/contribute.html.md
+++ b/asf-site-src/source/contribute.html.md
@@ -42,4 +42,4 @@ The naming of the patch should be in (JIRA).(patch number).patch or, if it needs
Below are guides meant to give you help accomplishing specific tasks for the project:
* [Maintaining the Yetus Website](website) - walks through how to view the website locally, update various static and generated pages, and render the html for publishing.
-* [Working with Release Candiates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts.
+* [Working with Release Candidates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts.
diff --git a/asf-site-src/source/contribute/releases.html.md b/asf-site-src/source/contribute/releases.html.md
index 288da4d..ed4da51 100644
--- a/asf-site-src/source/contribute/releases.html.md
+++ b/asf-site-src/source/contribute/releases.html.md
@@ -614,7 +614,7 @@ Once a release candidate obtains majority approval from the PMC, there are sever
1. Type the tag that you just pushed into the 'tag' box.
1. Use categories 'Code quality' and 'Continuous integration'
1. Release Title should reflect the version
-1. Describe this release should be a cut-down version of the anouncement email (drop SHA and direct download links. main page, github actions, and release notes should be mentioned)
+1. Describe this release should be a cut-down version of the announcement email (drop SHA and direct download links. main page, github actions, and release notes should be mentioned)
1. Mark 'This is a pre-release'
1. Verify everything looks good.
1. Publish release
diff --git a/asf-site-src/source/documentation/in-progress.html.md b/asf-site-src/source/documentation/in-progress.html.md
index 7046bdb..dac3a07 100644
--- a/asf-site-src/source/documentation/in-progress.html.md
+++ b/asf-site-src/source/documentation/in-progress.html.md
@@ -23,8 +23,8 @@ The [Yetus Precommit build, patch, and CI suite](precommit) allows projects to c
# Yetus Release Doc Maker
-The Release Documentation Maker allows projects to generate nicely formated Markdown Changelogs and Release Notes based upon JIRA. You can view that
-documenation [here](releasedocmaker). See also the [yetus-maven-plugin](#yetus-maven-plugin) for Apache Maven-specific information.
+The Release Documentation Maker allows projects to generate nicely formatted Markdown Changelogs and Release Notes based upon JIRA. You can view that
+documentation [here](releasedocmaker). See also the [yetus-maven-plugin](#yetus-maven-plugin) for Apache Maven-specific information.
# Yetus Shelldocs
diff --git a/asf-site-src/source/documentation/in-progress/precommit/buildtools.html.md b/asf-site-src/source/documentation/in-progress/precommit/buildtools.html.md
index 9e87cec..1cb270f 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/buildtools.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/buildtools.html.md
@@ -197,4 +197,4 @@ By default, test-patch will pass -Ptest-patch to Maven. This will allow you to c
Maven will test eclipse and site if maven is being used as the build tool and appropriate files trigger them.
-Maven will trigger add a maven install test when the `maven_add_install` function has been used and the related tests are requierd. Plug-ins that need to run maven before MUST call it as part of their respective initialize functions, otherwise maven may fail unexpectedly. All Apache Yetus provided plug-ins that require maven will trigger the maven install functionality.
+Maven will trigger add a maven install test when the `maven_add_install` function has been used and the related tests are required. Plug-ins that need to run maven before MUST call it as part of their respective initialize functions, otherwise maven may fail unexpectedly. All Apache Yetus provided plug-ins that require maven will trigger the maven install functionality.
diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
index 50ad1fa..c16af5c 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/docker-cleanup.html.md
@@ -29,7 +29,7 @@
# Usage
-Executing `docker-cleanup` will peform the same Docker image as the `--robot` option.
+Executing `docker-cleanup` will perform the same Docker image as the `--robot` option.
```bash
$ docker-cleanup
diff --git a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
index 131a503..592181c 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/docker.html.md
@@ -76,7 +76,7 @@ Additionally, if bash v4 and Linux is in use, a separate process is launched to
# Privileged Mode
-In some cases, the work being preformed inside the Docker container requires extra permissions. Using the `--docker-privd` option enables Docker's extended privileges to the container.
+In some cases, the work being performed inside the Docker container requires extra permissions. Using the `--docker-privd` option enables Docker's extended privileges to the container.
# Docker in Docker
diff --git a/precommit/src/main/shell/core.d/00-yetuslib.sh b/precommit/src/main/shell/core.d/00-yetuslib.sh
index 13754f8..1ef0607 100755
--- a/precommit/src/main/shell/core.d/00-yetuslib.sh
+++ b/precommit/src/main/shell/core.d/00-yetuslib.sh
@@ -255,7 +255,7 @@ function yetus_array_to_comma
}
## @description Convert a file to an array.
-## @description Comments on the beginning of the line and emtpy lines are stripped.
+## @description Comments on the beginning of the line and empty lines are stripped.
## @audience public
## @stability evolving
## @replaceable no
diff --git a/precommit/src/main/shell/core.d/01-common.sh b/precommit/src/main/shell/core.d/01-common.sh
index 0f37733..d8f2c87 100755
--- a/precommit/src/main/shell/core.d/01-common.sh
+++ b/precommit/src/main/shell/core.d/01-common.sh
@@ -852,7 +852,7 @@ function generate_stack
exit 1
}
-## @description remove entries from paramater tracker
+## @description remove entries from parameter tracker
## @audience public
## @stability stable
## @replaceable no
diff --git a/precommit/src/main/shell/core.d/docker.sh b/precommit/src/main/shell/core.d/docker.sh
index 4dbf656..82dccff 100755
--- a/precommit/src/main/shell/core.d/docker.sh
+++ b/precommit/src/main/shell/core.d/docker.sh
@@ -306,7 +306,7 @@ function dockercmd
"${DOCKERCMD}" "${subcmd}" "$@"
}
-## @description Convet docker's time format to ctime
+## @description Convert docker's time format to ctime
## @audience private
## @stability evolving
## @replaceable no
@@ -777,7 +777,7 @@ function docker_run_image
--tag "${patchimagename}" \
"${randir}" >/dev/null
retval=$?
- popd >/dev/null || retun 1
+ popd >/dev/null || return 1
rm -rf "${randir}"
diff --git a/precommit/src/main/shell/robots.d/jenkins.sh b/precommit/src/main/shell/robots.d/jenkins.sh
index 8a4c6b4..1be0bfb 100755
--- a/precommit/src/main/shell/robots.d/jenkins.sh
+++ b/precommit/src/main/shell/robots.d/jenkins.sh
@@ -32,7 +32,7 @@ if [[ -n "${JENKINS_URL}" && -n "${EXECUTOR_NUMBER}" ]] &&
USER_PARAMS+=("--empty-patch")
# if we are running in an Jenkins docker container
- # spawed by using agent, then there is a good chance
+ # spawned by using agent, then there is a good chance
# that the user running the test is in a bad state:
# no password entry, no home directory, etc.
if [[ "${HOME}" == / ]]; then
diff --git a/precommit/src/main/shell/test-patch-docker/Dockerfile b/precommit/src/main/shell/test-patch-docker/Dockerfile
index bb0f7d8..d5873f3 100644
--- a/precommit/src/main/shell/test-patch-docker/Dockerfile
+++ b/precommit/src/main/shell/test-patch-docker/Dockerfile
@@ -340,7 +340,7 @@ RUN curl -sSL \
####
# YETUS CUT HERE
-# Anthing after the above line is ignored by Yetus, so could
+# Anything after the above line is ignored by Yetus, so could
# include other requirements not needed by your development
# (but not build) environment
###
diff --git a/precommit/src/main/shell/test-patch.d/github.sh b/precommit/src/main/shell/test-patch.d/github.sh
index 1b10005..eb01f1e 100755
--- a/precommit/src/main/shell/test-patch.d/github.sh
+++ b/precommit/src/main/shell/test-patch.d/github.sh
@@ -147,7 +147,7 @@ function github_breakup_url
fi
}
-## @description initalize github
+## @description initialize github
function github_initialize
{
if [[ -z "${GITHUB_REPO}" ]]; then
@@ -733,7 +733,7 @@ function github_finalreport
# Customize the message based upon type of passing:
# - if there were no warnings, then just put a single message
- # - if there were warningss, but no logs, then just mention warnings
+ # - if there were warnings, but no logs, then just mention warnings
# - if there were warnings and logs, handle in the next section
if [[ "${result}" == 0 ]]; then
i=0
@@ -782,7 +782,7 @@ function github_finalreport
# give a separate status for each:
# - failure
# - failure w/log
- # - successs w/warning log
+ # - success w/warning log
i=0
until [[ ${i} -eq ${#TP_VOTE_TABLE[@]} ]]; do
diff --git a/precommit/src/main/shell/test-patch.sh b/precommit/src/main/shell/test-patch.sh
index f806515..f0cf777 100755
--- a/precommit/src/main/shell/test-patch.sh
+++ b/precommit/src/main/shell/test-patch.sh
@@ -1294,7 +1294,7 @@ function git_checkout
}
## @description Confirm the given branch is a git reference
-## @descriptoin or a valid gitXYZ commit hash
+## @description or a valid gitXYZ commit hash
## @audience private
## @stability evolving
## @replaceable no
@@ -2394,7 +2394,7 @@ function runtests
## @description Calculate the differences between the specified files
## @description using just the column+ messages (third+ column in a
-## @descriptoin colon delimated flie) and output it to stdout.
+## @description colon delimated file) and output it to stdout.
## @audience public
## @stability evolving
## @replaceable no
@@ -2439,7 +2439,7 @@ function column_calcdiffs
## @description Calculate the differences between the specified files
## @description using just the error messages (last column in a
-## @descriptoin colon delimated flie) and output it to stdout.
+## @description colon delimated file) and output it to stdout.
## @audience public
## @stability evolving
## @replaceable no
@@ -2516,7 +2516,7 @@ function calcdiffs
fi
}
-## @description generate a standarized calcdiff status message
+## @description generate a standardized calcdiff status message
## @audience public
## @stability evolving
## @replaceable no
diff --git a/releasedocmaker/src/main/python/releasedocmaker/__init__.py b/releasedocmaker/src/main/python/releasedocmaker/__init__.py
index 5778c06..16fa7ed 100755
--- a/releasedocmaker/src/main/python/releasedocmaker/__init__.py
+++ b/releasedocmaker/src/main/python/releasedocmaker/__init__.py
@@ -305,7 +305,7 @@ class Jira(object):
return self.incompat
def get_important(self):
- """ get importat flag """
+ """ get important flag """
if self.important is None:
field = self.parent.field_id_map['Flags']
self.important = False
diff --git a/releasedocmaker/src/main/python/releasedocmaker/utils.py b/releasedocmaker/src/main/python/releasedocmaker/utils.py
index ae57fd1..3964671 100755
--- a/releasedocmaker/src/main/python/releasedocmaker/utils.py
+++ b/releasedocmaker/src/main/python/releasedocmaker/utils.py
@@ -172,7 +172,7 @@ class Outputs(object):
self.others[key].write(pattern % both)
def write_key_raw(self, key, input_string):
- """ write everything withotu changes """
+ """ write everything without changes """
self.base.write(input_string)
if key in self.others:
self.others[key].write(input_string)
diff --git a/start-build-env.sh b/start-build-env.sh
index aeb1bff..8c192ca 100755
--- a/start-build-env.sh
+++ b/start-build-env.sh
@@ -117,7 +117,7 @@ if [[ ! -d ${HOME}/.gnupg ]]; then
fi
dockerargs+=(-v "${HOME}/.gnupg:/home/${USER_NAME}/.gnupg${V_OPTS:-}")
-# git opertions
+# git operations
if [[ ! -d ${HOME}/.ssh ]]; then
mkdir "${HOME}/.ssh"
fi
[yetus] 04/05: YETUS-1059. Github Status Recovery Tool (#184)
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 38c78a7806774a721ca5562326aabcb15cd978aa
Author: Allen Wittenauer <aw...@apache.org>
AuthorDate: Mon Nov 2 06:18:00 2020 -0800
YETUS-1059. Github Status Recovery Tool (#184)
---
Jenkinsfile | 28 ++-
.../precommit/github-status-recovery.html.md | 50 ++++
.../in-progress/precommit/index.html.md | 1 +
.../in-progress/precommit/robots/jenkins.html.md | 23 +-
precommit/src/main/shell/core.d/linecomments.sh | 132 +++++++++++
precommit/src/main/shell/github-status-recovery.sh | 188 +++++++++++++++
precommit/src/main/shell/test-patch.d/github.sh | 259 ++++++++++++++++++++-
precommit/src/main/shell/test-patch.sh | 114 +--------
8 files changed, 675 insertions(+), 120 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 0840c93..6a59fdc 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -104,9 +104,6 @@ pipeline {
YETUS_ARGS+=(--jira-password="${JIRA_PASSWORD}")
YETUS_ARGS+=(--jira-user="${JIRA_USER}")
- # disable per-line comments
- YETUS_ARGS+=(--linecomments='')
-
# pylint settings
YETUS_ARGS+=('--pylint=pylint2')
@@ -167,6 +164,31 @@ pipeline {
post {
always {
script {
+
+ // Publish status if it was missed
+ withCredentials([usernamePassword(credentialsId: 'apache-yetus-at-github.com',
+ passwordVariable: 'GITHUB_TOKEN',
+ usernameVariable: 'GITHUB_USER')]) {
+ sh '''#!/usr/bin/env bash
+
+ # enable writing back to Github
+ YETUS_ARGS+=(--github-token="${GITHUB_TOKEN}")
+ YETUS_ARGS+=("--patch-dir=${WORKSPACE}/${YETUS_RELATIVE_PATCHDIR}")
+
+ if [[ "${USE_DEBUG_FLAG}" == true ]]; then
+ YETUS_ARGS+=("--debug")
+ fi
+
+ # run test-patch from the source tree specified up above
+ TESTPATCHBIN=${WORKSPACE}/src/precommit/src/main/shell/github-status-recovery.sh
+
+ # execute! (we are using bash instead of the
+ # bin in case the perms get messed up)
+ /usr/bin/env bash "${TESTPATCHBIN}" "${YETUS_ARGS[@]}" ${EXTRA_ARGS} || true
+
+ '''
+ }
+
// Publish JUnit results
try {
junit "${env.YETUS_RELATIVE_PATCHDIR}/junit-report.xml"
diff --git a/asf-site-src/source/documentation/in-progress/precommit/github-status-recovery.html.md b/asf-site-src/source/documentation/in-progress/precommit/github-status-recovery.html.md
new file mode 100644
index 0000000..0cb6740
--- /dev/null
+++ b/asf-site-src/source/documentation/in-progress/precommit/github-status-recovery.html.md
@@ -0,0 +1,50 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# GitHub Status Recovery
+
+<!-- MarkdownTOC levels="1,2,3" autolink="true" indent=" " bullets="*" bracket="round" -->
+
+* [Problem Statement](#problem-statement)
+* [Usage](#usage)
+* [Disabling Annotations](#disabling-annotations)
+
+<!-- /MarkdownTOC -->
+
+# Problem Statement
+
+For CI systems that use GitHub outside of GitHub Actions, they may make available a GitHub Checks token.
+Unfortunately, as of this writing (2020-10-30), GitHub sets the expiry of such a token to 1 hour.
+For some users of Apache Yetus, their precommit job may take longer than one hour. In order to workaround
+this limitation, the `github-status-recovery` program may be used.
+
+# Usage
+
+The usage is relatively simple:
+
+```bash
+$ github-status-recovery --patch-dir=<pre-existing patch directory> --github-token=<token>
+```
+
+If the previous run of `test-patch` failed to write the status, `github-status-recovery` will
+re-process the saved JSON files as well as write GitHub Checks Annotations if they exist.
+
+# Disabling Annotations
+
+If for some reason you do not wish annotations to be written, they may be disabled with `--github-annotations=false`.
diff --git a/asf-site-src/source/documentation/in-progress/precommit/index.html.md b/asf-site-src/source/documentation/in-progress/precommit/index.html.md
index ed0f561..b9d3a9f 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/index.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/index.html.md
@@ -156,6 +156,7 @@ Language Support, Licensing, and more:
capacities without needing to use the full `test-patch` runtime:
* [docker-cleanup](docker-cleanup) - safe removal of Docker resources for multi-executor CI systems
+* [github-status-recovery](github-status-recovery) - Apache Yetus status on GitHub, even for long running jobs
* [jenkins-admin](admin) - Jenkins<->JIRA patch bridge
* [qbt](qbt) - Quality Build Tool, for branch-specific testing
* [smart-apply-patch](smart-apply-patch) - CLI manipulation and query of patch files, PRs, and more
diff --git a/asf-site-src/source/documentation/in-progress/precommit/robots/jenkins.html.md b/asf-site-src/source/documentation/in-progress/precommit/robots/jenkins.html.md
index 4d81947..3de1b8c 100644
--- a/asf-site-src/source/documentation/in-progress/precommit/robots/jenkins.html.md
+++ b/asf-site-src/source/documentation/in-progress/precommit/robots/jenkins.html.md
@@ -90,8 +90,27 @@ for the optimal `test-patch` experience. Configure up to the "Configuring the G
...
```
-Doing so will enable in many circumstances a bit more functionality, such as
-GitHub Statuses.
+Doing so will enable in many circumstances a bit more functionality, such as GitHub Statuses.
+
+## Enabling GitHub Status Recovery
+
+If the Apache Yetus section of your job typically runs longer than 1 hour and you use GitHub as the primary bugsystem,
+it is recommend to use the `github-status-recovery` utility in the `post` section of your `Jenkinsfile`. For example:
+
+```groovy
+ post {
+ always {
+ script {
+ // Publish status if it was missed
+ withCredentials([usernamePassword(credentialsId: github-app',
+ passwordVariable: 'GITHUB_TOKEN',
+ usernameVariable: 'GITHUB_USER')]) {
+ sh '''github-status-recovery --patch-dir="${PATCH_DIR}"" --github-token="${GITHUB_TOKEN}"'''
+ }
+ }
+ }
+ }
+```
See also:
diff --git a/precommit/src/main/shell/core.d/linecomments.sh b/precommit/src/main/shell/core.d/linecomments.sh
new file mode 100755
index 0000000..4ccd5e0
--- /dev/null
+++ b/precommit/src/main/shell/core.d/linecomments.sh
@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Make sure that bash version meets the pre-requisite
+
+if [[ -z "${BASH_VERSINFO[0]}" ]] \
+ || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \
+ || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then
+ echo "bash v3.2+ is required. Sorry."
+ exit 1
+fi
+
+## @description Queue up comments to write into bug systems
+## @description that have code review support, if such support
+## @description enabled/available.
+## @description File should be in the form of "file:line[:column]:comment"
+## @audience public
+## @stability evolving
+## @replaceable no
+## @param plugin
+## @param filename
+function bugsystem_linecomments_queue
+{
+ declare plugin=$1
+ declare fn=$2
+ declare line
+ declare linenum
+ declare text
+ declare columncheck
+ declare column
+ declare rol
+ declare file
+
+ if [[ -z "${BUGLINECOMMENTS}" ]]; then
+ return 0
+ fi
+
+ for line in "${VOTE_FILTER[@]}"; do
+ if [[ "${plugin}" == "${line}" ]]; then
+ return 0
+ fi
+ done
+
+ pushd "${BASEDIR}" >/dev/null || return 1
+ while read -r line; do
+ file=${line%%:*}
+
+ if [[ ! -e "${file}" ]]; then
+ continue
+ fi
+
+ rol=${line/#${file}:}
+ if [[ "${file}" =~ ^\./ ]]; then
+ file=${file:2}
+ fi
+
+ linenum=${rol%%:*}
+ rol=${rol/#${linenum}:}
+ columncheck=${rol%%:*}
+ if [[ "${columncheck}" =~ ^[0-9]+$ ]]; then
+ column=${columncheck}
+ text=${rol/#${column}:}
+ else
+ column="0"
+ text=${rol}
+ fi
+
+ echo "${file}:${linenum}:${column}:${plugin}:${text}" >> "${PATCH_DIR}/results-full.txt"
+
+ done < "${fn}"
+
+ popd >/dev/null || return 1
+
+}
+
+## @description Write all of the bugsystem linecomments
+## @audience public
+## @stability evolving
+## @replaceable no
+function bugsystem_linecomments_trigger
+{
+ declare plugin
+ declare fn
+ declare line
+ declare linenum
+ declare text
+ declare column
+
+ if [[ ! -f "${PATCH_DIR}/results-full.txt" ]]; then
+ return 0
+ fi
+
+ # sort the file such that all files and lines are now next to each other
+ sort -k1,1 -k2,2n -k3,3n -k4,4 "${PATCH_DIR}/results-full.txt" > "${PATCH_DIR}/linecomments-sorted.txt"
+ mv "${PATCH_DIR}/linecomments-sorted.txt" "${PATCH_DIR}/results-full.txt"
+
+ while read -r line;do
+ fn=${line%%:*}
+ rol=${line/#${fn}:}
+ linenum=${rol%%:*}
+ rol=${rol/#${linenum}:}
+ column=${rol%%:*}
+ rol=${rol/#${column}:}
+ plugin=${rol%%:*}
+ text=${rol/#${plugin}:}
+
+ for bugs in ${BUGLINECOMMENTS}; do
+ if declare -f "${bugs}_linecomments" >/dev/null;then
+ "${bugs}_linecomments" "${fn}" "${linenum}" "${column}" "${plugin}" "${text}"
+ fi
+ done
+ done < "${PATCH_DIR}/results-full.txt"
+
+ for bugs in ${BUGLINECOMMENTS}; do
+ if declare -f "${bugs}_linecomments_end" >/dev/null;then
+ "${bugs}_linecomments_end"
+ fi
+ done
+}
\ No newline at end of file
diff --git a/precommit/src/main/shell/github-status-recovery.sh b/precommit/src/main/shell/github-status-recovery.sh
new file mode 100755
index 0000000..97bf95a
--- /dev/null
+++ b/precommit/src/main/shell/github-status-recovery.sh
@@ -0,0 +1,188 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# no shelldocs required from this file
+# SHELLDOC-IGNORE
+
+# Make sure that bash version meets the pre-requisite
+
+if [[ -z "${BASH_VERSINFO[0]}" ]] \
+ || [[ "${BASH_VERSINFO[0]}" -lt 3 ]] \
+ || [[ "${BASH_VERSINFO[0]}" -eq 3 && "${BASH_VERSINFO[1]}" -lt 2 ]]; then
+ echo "bash v3.2+ is required. Sorry."
+ exit 1
+fi
+
+this="${BASH_SOURCE-$0}"
+BINDIR=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P)
+BINNAME=${this##*/}
+BINNAME=${BINNAME%.sh}
+#shellcheck disable=SC2034
+STARTINGDIR=$(pwd)
+#shellcheck disable=SC2034
+USER_PARAMS=("$@")
+#shellcheck disable=SC2034
+QATESTMODE=false
+#shellcheck disable=SC2034
+ISODATESTART=$(date +"%Y-%m-%dT%H:%M:%SZ")
+
+## @description import core library routines
+## @audience private
+## @stability evolving
+function import_core
+{
+ declare filename
+
+ for filename in "${BINDIR}/core.d"/*; do
+ # shellcheck disable=SC1091
+ # shellcheck source=core.d/01-common.sh
+ . "${filename}"
+ done
+}
+
+## @description import plugins then remove the stuff we don't need
+## @audience public
+## @stability stable
+## @replaceable no
+function import_including_github
+{
+ #shellcheck disable=SC2034
+ ENABLED_PLUGINS='github'
+ importplugins
+ yetus_debug "Removing BUILDTOOLS, TESTTYPES, and TESTFORMATS from installed plug-in list"
+ #shellcheck disable=SC2034
+ BUILDTOOLS=()
+ #shellcheck disable=SC2034
+ TESTTYPES=()
+ #shellcheck disable=SC2034
+ TESTFORMATS=()
+ #shellcheck disable=SC2034
+ BUGSYSTEMS=('github')
+ #shellcheck disable=SC2034
+ BUGLINECOMMENTS='github'
+ #shellcheck disable=SC2034
+ GITHUB_STATUS_RECOVER_TOOL=true
+ GITHUB_CHECK_ANNOTATIONS=true
+}
+
+## @description Setup the default global variables
+## @audience public
+## @stability stable
+## @replaceable no
+function setup_defaults
+{
+ common_defaults
+}
+
+## @description Interpret the command line parameters
+## @audience private
+## @stability stable
+## @replaceable no
+## @param $@
+## @return May exit on failure
+function parse_args
+{
+ declare i
+ common_args "$@"
+
+ for i in "$@"; do
+ case ${i} in
+ --github-annotations=*)
+ delete_parameter "${i}"
+ GITHUB_CHECK_ANNOTATIONS="${i#*=}"
+ ;;
+ esac
+ done
+}
+
+## @description Print the usage information
+## @audience public
+## @stability stable
+## @replaceable no
+function yetus_usage
+{
+ import_including_github
+ github_usage
+
+ echo "${BINNAME} [OPTIONS]"
+
+ yetus_add_option "--github-annotations=<bool>" "Enable GitHub Checks Annoations [default: ${GITHUB_CHECK_ANNOTATIONS}]"
+ yetus_add_option "--curl-cmd=<cmd>" "The 'curl' command to use (default 'curl')"
+ yetus_add_option "--debug" "If set, then output some extra stuff to stderr"
+ yetus_add_option "--grep-cmd=<cmd>" "The 'grep' command to use (default 'grep')"
+ yetus_add_option "--ignore-unknown-options=<bool>" "Continue despite unknown options (default: ${IGNORE_UNKNOWN_OPTIONS})"
+ yetus_add_option "--patch-dir=<dir>" "The directory for working and output files (default '/tmp/test-patch-${PROJECT_NAME}/pid')"
+
+ yetus_generic_columnprinter "${YETUS_OPTION_USAGE[@]}"
+ yetus_reset_usage
+}
+
+## @description Large display for the user console
+## @audience public
+## @stability stable
+## @replaceable no
+## @param string
+## @return large chunk of text
+function big_console_header
+{
+ local text="$*"
+ local spacing=$(( (75+${#text}) /2 ))
+ printf '\n\n'
+ echo "============================================================================"
+ echo "============================================================================"
+ printf '%*s\n' ${spacing} "${text}"
+ echo "============================================================================"
+ echo "============================================================================"
+ printf '\n\n'
+}
+
+## @description setup the parameter tracker for param errors
+## @audience private
+## @stability evolving
+function setup_parameter_tracker
+{
+ declare i
+
+ for i in "${USER_PARAMS[@]}"; do
+ if [[ "${i}" =~ ^-- ]]; then
+ i=${i%=*}
+ PARAMETER_TRACKER+=("${i}")
+ fi
+ done
+}
+
+trap "cleanup_and_exit 1" HUP INT QUIT TERM
+
+setup_parameter_tracker
+
+import_core
+
+setup_defaults
+
+parse_args "$@"
+
+import_including_github
+
+parse_args_plugins "$@"
+
+if [[ "${#PARAMETER_TRACKER}" -gt 0 ]]; then
+ yetus_error "ERROR: Unprocessed flag(s): ${PARAMETER_TRACKER[*]}"
+ if [[ "${IGNORE_UNKNOWN_OPTIONS}" == false ]]; then
+ cleanup_and_exit 1
+ fi
+fi
+
+github_status_recovery
diff --git a/precommit/src/main/shell/test-patch.d/github.sh b/precommit/src/main/shell/test-patch.d/github.sh
index eb01f1e..cdfec87 100755
--- a/precommit/src/main/shell/test-patch.d/github.sh
+++ b/precommit/src/main/shell/test-patch.d/github.sh
@@ -36,6 +36,8 @@ GITHUB_REPO=""
GITHUB_TOKEN="${GITHUB_TOKEN-}"
GITHUB_ISSUE=""
GITHUB_USE_EMOJI_VOTE=false
+GITHUB_STATUS_RECOVERY_COUNTER=1
+GITHUB_STATUS_RECOVER_TOOL=false
declare -a GITHUB_AUTH
# private globals...
@@ -47,7 +49,9 @@ function github_usage
yetus_add_option "--github-base-url=<url>" "The URL of the github server (default:'${GITHUB_BASE_URL}')"
yetus_add_option "--github-repo=<repo>" "github repo to use (default:'${GITHUB_REPO}')"
yetus_add_option "--github-token=<token>" "The token to use to read/write to github"
- yetus_add_option "--github-use-emoji-vote" "Whether to use emoji to represent the vote result on github [default: ${GITHUB_USE_EMOJI_VOTE}]"
+ if [[ "${GITHUB_STATUS_RECOVER_TOOL}" == false ]]; then
+ yetus_add_option "--github-use-emoji-vote" "Whether to use emoji to represent the vote result on github [default: ${GITHUB_USE_EMOJI_VOTE}]"
+ fi
}
function github_parse_args
@@ -454,6 +458,129 @@ function github_locate_patch
esac
}
+## @description Generate a Github Check Run ID
+## @stability evolving
+## @audience private
+function github_start_checkrun
+{
+ declare tempfile="${PATCH_DIR}/ghcheckrun.$$.${RANDOM}"
+ declare output="${PATCH_DIR}/ghcheckrun.json"
+
+ if [[ -z "${GITHUB_SHA}" ]]; then
+ GITHUB_SHA=$("${GREP}" \"sha\" "${PATCH_DIR}/github-pull.json" 2>/dev/null \
+ | head -1 \
+ | cut -f4 -d\")
+ fi
+
+ if [[ -z "${GITHUB_SHA}" ]]; then
+ return 0
+ fi
+
+ # don't need this under GHA
+ if [[ "${ROBOTTYPE}" == 'githubactions' ]]; then
+ return 0
+ fi
+
+ if [[ "${OFFLINE}" == true ]]; then
+ return 0
+ fi
+
+ if [[ "${#GITHUB_AUTH[@]}" -eq 0 ]]; then
+ return 0
+ fi
+
+ {
+ printf "{"
+ echo "\"name\":\"Apache Yetus ${ROBOTTYPE}\","
+ echo "\"head_sha\": \"${GITHUB_SHA}\","
+ echo "\"details_url\": \"${BUILD_URL}${BUILD_URL_CONSOLE}\","
+ echo "\"external_id\": \"${INSTANCE}\","
+ echo "\"status\": \"in_progress\","
+ echo "\"started_at\": \"${ISODATESTART}\""
+ # external_id instance_id?
+ # status queued, in_progress, completed
+ # started_at ISO-8601
+ # conclusion , required for status=completed, completed_at=value
+ # success, failure, neutral, cancelled, skipped, timed_out, action_required
+ # completed_at ISO-8601
+ # output see github docs @ https://docs.github.com/en/rest/reference/checks#update-a-check-run
+ echo "}"
+ } > "${tempfile}"
+
+ "${CURL}" --silent --fail -X POST \
+ -H "Accept: application/vnd.github.antiope-preview+json" \
+ -H "Content-Type: application/json" \
+ "${GITHUB_AUTH[@]}" \
+ -d @"${tempfile}" \
+ --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/check-runs" \
+ --output "${output}" 2>/dev/null
+
+ GITHUB_CHECK_RUN_ID=$("${GREP}" \"id\" "${output}" 2>/dev/null \
+ | head -1 \
+ | cut -f2 -d: \
+ | cut -f1 -d,)
+ GITHUB_CHECK_RUN_ID=${GITHUB_CHECK_RUN_ID// /}
+}
+
+## @description Generate a Github Check Run ID
+## @stability evolving
+## @audience private
+function github_end_checkrun
+{
+ declare result=$1
+ declare tempfile="${PATCH_DIR}/ghcheckrun.$$.${RANDOM}"
+ declare output="${PATCH_DIR}/ghcheckrun-final.json"
+ declare conclusion
+
+ # don't need this under GHA
+ if [[ "${ROBOTTYPE}" == 'githubactions' ]]; then
+ return 0
+ fi
+
+ if [[ "${OFFLINE}" == true ]]; then
+ return 0
+ fi
+
+ if [[ "${#GITHUB_AUTH[@]}" -eq 0 ]]; then
+ return 0
+ fi
+
+ if [[ "${result}" -eq 0 ]]; then
+ conclusion="success"
+ else
+ conclusion="failure"
+ fi
+
+ finishdate=$(date +"%Y-%m-%dT%H:%M:%SZ")
+
+ {
+ printf "{"
+ echo "\"conclusion\":\"${conclusion}\","
+ echo "\"status\": \"completed\","
+ echo "\"completed_at\": \"${finishdate}\""
+ echo "}"
+ } > "${tempfile}"
+
+ "${CURL}" --fail --silent -X PATCH \
+ -H "Accept: application/vnd.github.antiope-preview+json" \
+ -H "Content-Type: application/json" \
+ "${GITHUB_AUTH[@]}" \
+ -d @"${tempfile}" \
+ --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/check-runs/${GITHUB_CHECK_RUN_ID}" \
+ --output "${output}" 2>/dev/null
+ rm "${tempfile}"
+}
+
+## @description Write a Github Checks Annotation
+## @param filename
+## @param linenum
+## @param column
+## @param plugin
+## @param text
+## @stability evolving
+## @audience private
function github_linecomments
{
declare file=$1
@@ -462,6 +589,10 @@ function github_linecomments
declare plugin=$4
shift 4
declare text=$*
+ declare tempfile="${PATCH_DIR}/ghcomment.$$.${RANDOM}"
+ declare header
+ declare -a linehandler
+ declare -a colhandler
if [[ "${ROBOTTYPE}" == 'githubactions' ]]; then
if [[ -z "${column}" ]] || [[ "${column}" == 0 ]]; then
@@ -471,6 +602,69 @@ function github_linecomments
fi
return 0
fi
+
+ if [[ "${OFFLINE}" == true ]]; then
+ echo "Github Plugin: Running in offline, comment skipped."
+ return 0
+ fi
+
+ if [[ "${#GITHUB_AUTH[@]}" -eq 0 ]]; then
+ return 0
+ fi
+
+ if [[ -z "${GITHUB_CHECK_RUN_ID}" ]]; then
+ if ! github_start_checkrun; then
+ yetus_error "ERROR: Cannot generate a Github Check Run ID"
+ return 1
+ fi
+ fi
+
+ linehandler=(\"start_line\": "${linenum},")
+ linehandler+=(\"end_line\": "${linenum},")
+
+ if [[ -z "${column}" ]] || [[ "${column}" == 0 ]]; then
+ colhandler=()
+ else
+ colhandler=(\"start_column\": "${column},")
+ colhandler+=(\"end_column\": "${column},")
+ fi
+
+ newtext=$(echo "${text[*]}" | "${SED}" -e 's,\\,\\\\,g' \
+ -e 's,\",\\\",g' \
+ -e 's,$,\\r\\n,g' \
+ | tr -d '\n')
+
+ if [[ "${ROBOTTYPE}" ]]; then
+ header="Apache Yetus(${ROBOTTYPE})"
+ else
+ header="Apache Yetus"
+ fi
+
+ cat <<EOF > "${tempfile}"
+{
+ "output": {
+ "title": "${header}",
+ "summary": "Precommit Problem",
+ "annotations" : [{
+ "path": "${file}",
+ ${linehandler[@]}
+ ${colhandler[@]}
+ "annotation_level": "failure",
+ "message": "${newtext}"
+ }]
+ }
+}
+EOF
+
+ "${CURL}" --fail -X PATCH \
+ -H "Accept: application/vnd.github.antiope-preview+json" \
+ -H "Content-Type: application/json" \
+ "${GITHUB_AUTH[@]}" \
+ -d @"${tempfile}" \
+ --silent --location \
+ "${GITHUB_API_URL}/repos/${GITHUB_REPO}/check-runs/${GITHUB_CHECK_RUN_ID}" \
+ 2>/dev/null
+ rm "${tempfile}"
}
## @description Write the contents of a file to github
@@ -663,10 +857,15 @@ function github_write_comment
## @stability evolving
## @replaceable no
## @param runresult
-function github_status_write()
+function github_status_write
{
declare filename=$1
declare retval=0
+ declare recoverydir="${PATCH_DIR}/github-status-retry/${GITHUB_REPO}/${GIT_BRANCH_SHA}"
+
+ if [[ "${OFFLINE}" == true ]]; then
+ return 0
+ fi
if [[ "${#GITHUB_AUTH[@]}" -lt 1 ]]; then
echo "Github Plugin: no credentials provided to write a status."
@@ -684,11 +883,59 @@ function github_status_write()
retval=$?
if [[ ${retval} -gt 0 ]]; then
- echo "githubstatus-report: Failed to write status. Maybe the credential does not have write access to repo:status."
+ yetus_error "ERROR: Failed to write github status. Token expired or missing repo:status write?"
+ if [[ "${GITHUB_STATUS_RECOVER_TOOL}" == false ]]; then
+ mkdir -p "${recoverydir}"
+ cp -p "${tempfile}" "${recoverydir}/${GITHUB_STATUS_RECOVERY_COUNTER}.json"
+ ((GITHUB_STATUS_RECOVERY_COUNTER=GITHUB_STATUS_RECOVERY_COUNTER+1))
+ fi
fi
return ${retval}
}
+## @description Write a github status
+## @audience private
+## @stability evolving
+## @replaceable no
+## @param runresult
+function github_status_recovery
+{
+ declare filename
+ declare retval=0
+ declare retrydir
+
+ # get the first filename
+ filename=$(find "${PATCH_DIR}/github-status-retry" -type f -name '1.json' 2>/dev/null)
+
+ if [[ -z "${filename}" ]]; then
+ echo "No retry directory found in ${PATCH_DIR}. Maybe it was successful?"
+ return 0
+ fi
+
+ retrydir="${filename##*/github-status-retry/}"
+ GITHUB_REPO=$(echo "${retrydir}" | cut -f1-2 -d/)
+ GIT_BRANCH_SHA=$(echo "${retrydir}" | cut -f3 -d/)
+
+
+ github_initialize
+
+ if [[ "${#GITHUB_AUTH[@]}" -lt 1 ]]; then
+ echo "Github Plugin: no credentials provided to write a status."
+ return 1
+ fi
+
+ while read -r; do
+ github_status_write "${REPLY}"
+ retval=$?
+ done < <(find "${PATCH_DIR}/github-status-retry" -type f)
+
+ if [[ "${GITHUB_CHECK_ANNOTATIONS}" == true ]]; then
+ bugsystem_linecomments_trigger
+ fi
+
+ return ${retval}
+}
+
## @description Print out the finished details to the Github PR
## @audience private
## @stability evolving
@@ -711,6 +958,10 @@ function github_finalreport
declare -i i=0
declare header
+ if [[ "${OFFLINE}" == true ]]; then
+ return 0
+ fi
+
big_console_header "Adding GitHub Statuses"
if [[ "${#GITHUB_AUTH[@]}" -lt 1 ]]; then
@@ -723,6 +974,8 @@ function github_finalreport
return 0
fi
+ github_end_checkrun "${result}"
+
url=$(get_artifact_url)
if [[ "${ROBOTTYPE}" ]]; then
diff --git a/precommit/src/main/shell/test-patch.sh b/precommit/src/main/shell/test-patch.sh
index f0cf777..93d2e12 100755
--- a/precommit/src/main/shell/test-patch.sh
+++ b/precommit/src/main/shell/test-patch.sh
@@ -68,6 +68,8 @@ function setup_defaults
common_defaults
GLOBALTIMER=$("${AWK}" 'BEGIN {srand(); print srand()}')
+ # shellcheck disable=SC2034
+ ISODATESTART=$(date +"%Y-%m-%dT%H:%M:%SZ")
set_yetus_version
@@ -2177,118 +2179,6 @@ function check_unittests
return 0
}
-## @description Queue up comments to write into bug systems
-## @description that have code review support, if such support
-## @description enabled/available.
-## @description File should be in the form of "file:line[:column]:comment"
-## @audience public
-## @stability evolving
-## @replaceable no
-## @param plugin
-## @param filename
-function bugsystem_linecomments_queue
-{
- declare plugin=$1
- declare fn=$2
- declare line
- declare linenum
- declare text
- declare columncheck
- declare column
- declare rol
- declare file
-
- if [[ -z "${BUGLINECOMMENTS}" ]]; then
- return 0
- fi
-
- for line in "${VOTE_FILTER[@]}"; do
- if [[ "${plugin}" == "${line}" ]]; then
- return 0
- fi
- done
-
- pushd "${BASEDIR}" >/dev/null || return 1
- while read -r line; do
- file=${line%%:*}
-
- if [[ ! -e "${file}" ]]; then
- continue
- fi
-
- rol=${line/#${file}:}
- if [[ "${file}" =~ ^\./ ]]; then
- file=${file:2}
- fi
-
- linenum=${rol%%:*}
- rol=${rol/#${linenum}:}
- columncheck=${rol%%:*}
- if [[ "${columncheck}" =~ ^[0-9]+$ ]]; then
- column=${columncheck}
- text=${rol/#${column}:}
- else
- column="0"
- text=${rol}
- fi
-
- echo "${file}:${linenum}:${column}:${plugin}:${text}" >> "${PATCH_DIR}/linecomments-in.txt"
-
- done < "${fn}"
-
- popd >/dev/null || return 1
-
-}
-
-## @description Write all of the bugsystem linecomments
-## @audience public
-## @stability evolving
-## @replaceable no
-function bugsystem_linecomments_trigger
-{
- declare plugin
- declare fn
- declare line
- declare linenum
- declare text
- declare column
-
- if [[ ! -f "${PATCH_DIR}/linecomments-in.txt" ]]; then
- return 0
- fi
-
- # sort the file such that all files and lines are now next to each other
- sort -k1,1 -k2,2n -k3,3n -k4,4 "${PATCH_DIR}/linecomments-in.txt" > "${PATCH_DIR}/linecomments-sorted.txt"
-
- while read -r line;do
- fn=${line%%:*}
- rol=${line/#${fn}:}
- linenum=${rol%%:*}
- rol=${rol/#${linenum}:}
- column=${rol%%:*}
- rol=${rol/#${column}:}
- plugin=${rol%%:*}
- text=${rol/#${plugin}:}
-
- for bugs in ${BUGLINECOMMENTS}; do
- if declare -f "${bugs}_linecomments" >/dev/null;then
- "${bugs}_linecomments" "${fn}" "${linenum}" "${column}" "${plugin}" "${text}"
- fi
- done
- done < "${PATCH_DIR}/linecomments-sorted.txt"
-
- for bugs in ${BUGLINECOMMENTS}; do
- if declare -f "${bugs}_linecomments_end" >/dev/null;then
- "${bugs}_linecomments_end"
- fi
- done
-
- if [[ "${YETUS_SHELL_SCRIPT_DEBUG}" = true ]]; then
- yetus_debug "Keeping linecomments files for debugging"
- else
- rm "${PATCH_DIR}/linecomments-in.txt" "${PATCH_DIR}/linecomments-sorted.txt"
- fi
-}
## @description Write the final output to the selected bug system
## @audience private
[yetus] 03/05: YETUS-1067. mvn site fails on empty release (#189)
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aw pushed a commit to branch YETUS-1034-release
in repository https://gitbox.apache.org/repos/asf/yetus.git
commit 97b10e46e53831407458ee14925027cede88ae28
Author: Allen Wittenauer <aw...@apache.org>
AuthorDate: Mon Nov 2 06:17:46 2020 -0800
YETUS-1067. mvn site fails on empty release (#189)
---
asf-site-src/pom.xml | 1 +
releasedocmaker/pom.xml | 1 +
shelldocs/pom.xml | 6 +++---
3 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/asf-site-src/pom.xml b/asf-site-src/pom.xml
index fad41d9..806448f 100644
--- a/asf-site-src/pom.xml
+++ b/asf-site-src/pom.xml
@@ -154,6 +154,7 @@
<argument>--usetoday</argument>
<argument>--version</argument>
<argument>${project.version}</argument>
+ <argument>--empty</argument>
</arguments>
</configuration>
</execution>
diff --git a/releasedocmaker/pom.xml b/releasedocmaker/pom.xml
index cb09b2a..8d19124 100644
--- a/releasedocmaker/pom.xml
+++ b/releasedocmaker/pom.xml
@@ -34,6 +34,7 @@
<packaging>pom</packaging>
<build>
+
<resources>
<resource>
<directory>src/main/python</directory>
diff --git a/shelldocs/pom.xml b/shelldocs/pom.xml
index e0b5654..0876c69 100644
--- a/shelldocs/pom.xml
+++ b/shelldocs/pom.xml
@@ -35,9 +35,9 @@
<build>
- <resources>
- <resource>
- <directory>src/main/python</directory>
+ <resources>
+ <resource>
+ <directory>src/main/python</directory>
</resource>
</resources>