You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/01/29 06:59:16 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #3944: HADOOP-18098. Basic verification of release candidates
virajjasani opened a new pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944
### Description of PR
We should provide script for the basic sanity of Hadoop release candidates. It should include:
- Signature
- Checksum
- Rat check
- Build from src
- Build tarball from src
Although we can include unit test as well, but overall unit test run is going to be significantly higher, and precommit Jenkins builds provide better view of UT sanity.
### How was this patch tested?
Locally
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1029711951
Some sample results for the latest RCs:
After cleaning my local .m2 cache, I saw build failing due to the known issue as discussed on the VOTE thread.
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC20/ -P native -P yarn-ui
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_301): failed
- mvn clean install -DskipTests -P native -P yarn-ui
* Built tar from source (1.8.0_301): failed
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
```
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC3/
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_301): failed
- mvn clean install -DskipTests
* Built tar from source (1.8.0_301): failed
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
```
@jojochuang @aajisaka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
jojochuang commented on a change in pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#discussion_r796325710
##########
File path: dev-support/hadoop-vote.sh
##########
@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This script is useful to perform basic sanity tests for the given
+# Hadoop RC. It checks for the Checksum, Signature, Rat check,
+# Build from source and building tarball from the source.
+
+set -e -o pipefail
+
+usage() {
+ SCRIPT=$(basename "${BASH_SOURCE[@]}")
+
+ cat << __EOF
+hadoop-vote. A script for standard vote which verifies the following items
+1. Checksum of sources and binaries
+2. Signature of sources and binaries
+3. Rat check
+4. Built from source
+5. Built tar from source
+
+Usage: ${SCRIPT} -s | --source <url> [-k | --key <signature>] [-f | --keys-file-url <url>] [-o | --output-dir </path/to/use>] [-D property[=value]]
+ ${SCRIPT} -h | --help
+
+ -h | --help Show this screen.
+ -s | --source '<url>' A URL pointing to the release candidate sources and binaries
+ e.g. https://dist.apache.org/repos/dist/dev/hadoop/hadoop-<version>RC0/
+ -k | --key '<signature>' A signature of the public key, e.g. 9AD2AE49
+ -f | --keys-file-url '<url>' the URL of the key file, default is
+ https://downloads.apache.org/hadoop/common/KEYS
+ -o | --output-dir '</path>' directory which has the stdout and stderr of each verification target
+ -D | list of maven properties to set for the mvn invocations, i.e. <-D hbase.profile=2.0 -D skipTests> Defaults to unset
+__EOF
+}
+
+MVN_PROPERTIES=()
+
+while ((${#})); do
+ case "${1}" in
+ -h | --help )
+ usage; exit 0 ;;
+ -s | --source )
+ SOURCE_URL="${2}"; shift 2 ;;
+ -k | --key )
+ SIGNING_KEY="${2}"; shift 2 ;;
+ -f | --keys-file-url )
+ KEY_FILE_URL="${2}"; shift 2 ;;
+ -o | --output-dir )
+ OUTPUT_DIR="${2}"; shift 2 ;;
+ -D )
+ MVN_PROPERTIES+=("-D ${2}"); shift 2 ;;
+ * )
+ usage >&2; exit 1 ;;
+ esac
+done
+
+# Source url must be provided
+if [ -z "${SOURCE_URL}" ]; then
+ usage;
+ exit 1
+fi
+
+cat << __EOF
+Although This tool helps verifying Hadoop RC build and unit tests,
+operator may still consider verifying the following manually:
+1. Verify the API compatibility report
+2. Integration/performance/benchmark tests
+3. Object store specific Integration tests against an endpoint
+4. Verify overall unit test stability from Jenkins builds or locally
+5. Other concerns if any
+__EOF
+
+[[ "${SOURCE_URL}" != */ ]] && SOURCE_URL="${SOURCE_URL}/"
+HADOOP_RC_VERSION=$(tr "/" "\n" <<< "${SOURCE_URL}" | tail -n2)
+HADOOP_VERSION=$(echo "${HADOOP_RC_VERSION}" | sed -e 's/-RC[0-9]//g' | sed -e 's/hadoop-//g')
+JAVA_VERSION=$(java -version 2>&1 | cut -f3 -d' ' | head -n1 | sed -e 's/"//g')
+OUTPUT_DIR="${OUTPUT_DIR:-$(pwd)}"
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+ echo "Output directory ${OUTPUT_DIR} does not exist, please create it before running this script."
+ exit 1
+fi
+
+OUTPUT_PATH_PREFIX="${OUTPUT_DIR}"/"${HADOOP_RC_VERSION}"
+
+# default value for verification targets, 0 = failed
+SIGNATURE_PASSED=0
+CHECKSUM_PASSED=0
+RAT_CHECK_PASSED=0
+BUILD_FROM_SOURCE_PASSED=0
+BUILD_TAR_FROM_SOURCE_PASSED=0
+
+function download_and_import_keys() {
+ KEY_FILE_URL="${KEY_FILE_URL:-https://downloads.apache.org/hadoop/common/KEYS}"
+ echo "Obtain and import the publisher key(s) from ${KEY_FILE_URL}"
+ # download the keys file into file KEYS
+ wget -O KEYS "${KEY_FILE_URL}"
+ gpg --import KEYS
+ if [ -n "${SIGNING_KEY}" ]; then
+ gpg --list-keys "${SIGNING_KEY}"
+ fi
+}
+
+function download_release_candidate () {
+ # get all files from release candidate repo
+ wget -r -np -N -nH --cut-dirs 4 "${SOURCE_URL}"
+}
+
+function verify_signatures() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_verify_signatures
+ for file in *.tar.gz; do
+ gpg --verify "${file}".asc "${file}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_signatures && SIGNATURE_PASSED=1 || SIGNATURE_PASSED=0
+ done
+}
+
+function verify_checksums() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_verify_checksums
+ SHA_EXT=$(find . -name "*.sha*" | awk -F '.' '{ print $NF }' | head -n 1)
+ for file in *.tar.gz; do
+ sha512sum --tag "${file}" > "${file}"."${SHA_EXT}".tmp
+ diff "${file}"."${SHA_EXT}".tmp "${file}"."${SHA_EXT}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_checksums && CHECKSUM_PASSED=1 || CHECKSUM_PASSED=0
+ rm -f "${file}"."${SHA_EXT}".tmp
+ done
+}
+
+function unzip_from_source() {
+ tar -zxvf hadoop-"${HADOOP_VERSION}"-src.tar.gz
+ cd hadoop-"${HADOOP_VERSION}"-src
+}
+
+function rat_test() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_rat_test
+ mvn clean apache-rat:check "${MVN_PROPERTIES[@]}" 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_rat_test && RAT_CHECK_PASSED=1
+}
+
+function build_from_source() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_build_from_source
+ # No unit test run.
+ mvn clean install "${MVN_PROPERTIES[@]}" -DskipTests 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_build_from_source && BUILD_FROM_SOURCE_PASSED=1
Review comment:
it would be great to build with additional profiles:
-Pyarn-ui as well as -Pnative to build native code (ok, maybe optional)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1026856935
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 47s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 0s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 19m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 79m 3s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3944 |
| Optional Tests | dupname asflicense mvnsite unit codespell shellcheck shelldocs |
| uname | Linux f232d752fb5a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3e7228d05f50f8cb5a5e308f52892234145e6b37 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/testReport/ |
| Max. process+thread count | 683 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1029831896
I think this is good... Thanks Viraj. Do we have to wait for the next RC before merging this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3944: HADOOP-18098. Basic verification of release candidates
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1024854621
For 3.3.2RC3, here is the sample command:
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC2/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3944: HADOOP-18098. Basic verification of release candidates
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1024854621
For 3.3.2RC3, here is the sample command:
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC3/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1029833511
I believe we are good to merge this, just in case some minor bug is encountered later, I will create addendum for sure.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
jojochuang merged pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1029711951
Some sample results for the latest RCs:
After cleaning my local .m2 cache, I saw build failing due to the known issue as discussed on the VOTE thread.
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC20/ -P native -P yarn-ui
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_301): failed
- mvn clean install -DskipTests -P native -P yarn-ui
* Built tar from source (1.8.0_301): failed
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
```
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC3/
* Signature: ok
* Checksum : ok
* Rat check (1.8.0_301): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_301): failed
- mvn clean install -DskipTests
* Built tar from source (1.8.0_301): failed
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
```
Once we have the new RC, I will use this script to provide the basic verification of the RC.
@jojochuang @aajisaka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1029833511
I believe we are good to merge this and we can directly test for successful result when new RC lands. Just in case some minor bug is encountered later, I will create addendum for sure.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3944: HADOOP-18098. Basic verification of release candidates
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1024865270
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 0s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 19m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 84m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3944 |
| Optional Tests | dupname asflicense mvnsite unit codespell shellcheck shelldocs |
| uname | Linux 85b8e4564dce 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e89a84f2e7230243f1c07a933ce087cd17c07728 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/testReport/ |
| Max. process+thread count | 676 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3944: HADOOP-18098. Basic verification of release candidates
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#issuecomment-1024854621
For 3.3.2RC3, here is the sample command:
```
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC2/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3944: HADOOP-18098. Basic verification for the release candidate vote
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #3944:
URL: https://github.com/apache/hadoop/pull/3944#discussion_r796541446
##########
File path: dev-support/hadoop-vote.sh
##########
@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This script is useful to perform basic sanity tests for the given
+# Hadoop RC. It checks for the Checksum, Signature, Rat check,
+# Build from source and building tarball from the source.
+
+set -e -o pipefail
+
+usage() {
+ SCRIPT=$(basename "${BASH_SOURCE[@]}")
+
+ cat << __EOF
+hadoop-vote. A script for standard vote which verifies the following items
+1. Checksum of sources and binaries
+2. Signature of sources and binaries
+3. Rat check
+4. Built from source
+5. Built tar from source
+
+Usage: ${SCRIPT} -s | --source <url> [-k | --key <signature>] [-f | --keys-file-url <url>] [-o | --output-dir </path/to/use>] [-D property[=value]]
+ ${SCRIPT} -h | --help
+
+ -h | --help Show this screen.
+ -s | --source '<url>' A URL pointing to the release candidate sources and binaries
+ e.g. https://dist.apache.org/repos/dist/dev/hadoop/hadoop-<version>RC0/
+ -k | --key '<signature>' A signature of the public key, e.g. 9AD2AE49
+ -f | --keys-file-url '<url>' the URL of the key file, default is
+ https://downloads.apache.org/hadoop/common/KEYS
+ -o | --output-dir '</path>' directory which has the stdout and stderr of each verification target
+ -D | list of maven properties to set for the mvn invocations, i.e. <-D hbase.profile=2.0 -D skipTests> Defaults to unset
+__EOF
+}
+
+MVN_PROPERTIES=()
+
+while ((${#})); do
+ case "${1}" in
+ -h | --help )
+ usage; exit 0 ;;
+ -s | --source )
+ SOURCE_URL="${2}"; shift 2 ;;
+ -k | --key )
+ SIGNING_KEY="${2}"; shift 2 ;;
+ -f | --keys-file-url )
+ KEY_FILE_URL="${2}"; shift 2 ;;
+ -o | --output-dir )
+ OUTPUT_DIR="${2}"; shift 2 ;;
+ -D )
+ MVN_PROPERTIES+=("-D ${2}"); shift 2 ;;
+ * )
+ usage >&2; exit 1 ;;
+ esac
+done
+
+# Source url must be provided
+if [ -z "${SOURCE_URL}" ]; then
+ usage;
+ exit 1
+fi
+
+cat << __EOF
+Although This tool helps verifying Hadoop RC build and unit tests,
+operator may still consider verifying the following manually:
+1. Verify the API compatibility report
+2. Integration/performance/benchmark tests
+3. Object store specific Integration tests against an endpoint
+4. Verify overall unit test stability from Jenkins builds or locally
+5. Other concerns if any
+__EOF
+
+[[ "${SOURCE_URL}" != */ ]] && SOURCE_URL="${SOURCE_URL}/"
+HADOOP_RC_VERSION=$(tr "/" "\n" <<< "${SOURCE_URL}" | tail -n2)
+HADOOP_VERSION=$(echo "${HADOOP_RC_VERSION}" | sed -e 's/-RC[0-9]//g' | sed -e 's/hadoop-//g')
+JAVA_VERSION=$(java -version 2>&1 | cut -f3 -d' ' | head -n1 | sed -e 's/"//g')
+OUTPUT_DIR="${OUTPUT_DIR:-$(pwd)}"
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+ echo "Output directory ${OUTPUT_DIR} does not exist, please create it before running this script."
+ exit 1
+fi
+
+OUTPUT_PATH_PREFIX="${OUTPUT_DIR}"/"${HADOOP_RC_VERSION}"
+
+# default value for verification targets, 0 = failed
+SIGNATURE_PASSED=0
+CHECKSUM_PASSED=0
+RAT_CHECK_PASSED=0
+BUILD_FROM_SOURCE_PASSED=0
+BUILD_TAR_FROM_SOURCE_PASSED=0
+
+function download_and_import_keys() {
+ KEY_FILE_URL="${KEY_FILE_URL:-https://downloads.apache.org/hadoop/common/KEYS}"
+ echo "Obtain and import the publisher key(s) from ${KEY_FILE_URL}"
+ # download the keys file into file KEYS
+ wget -O KEYS "${KEY_FILE_URL}"
+ gpg --import KEYS
+ if [ -n "${SIGNING_KEY}" ]; then
+ gpg --list-keys "${SIGNING_KEY}"
+ fi
+}
+
+function download_release_candidate () {
+ # get all files from release candidate repo
+ wget -r -np -N -nH --cut-dirs 4 "${SOURCE_URL}"
+}
+
+function verify_signatures() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_verify_signatures
+ for file in *.tar.gz; do
+ gpg --verify "${file}".asc "${file}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_signatures && SIGNATURE_PASSED=1 || SIGNATURE_PASSED=0
+ done
+}
+
+function verify_checksums() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_verify_checksums
+ SHA_EXT=$(find . -name "*.sha*" | awk -F '.' '{ print $NF }' | head -n 1)
+ for file in *.tar.gz; do
+ sha512sum --tag "${file}" > "${file}"."${SHA_EXT}".tmp
+ diff "${file}"."${SHA_EXT}".tmp "${file}"."${SHA_EXT}" 2>&1 | tee -a "${OUTPUT_PATH_PREFIX}"_verify_checksums && CHECKSUM_PASSED=1 || CHECKSUM_PASSED=0
+ rm -f "${file}"."${SHA_EXT}".tmp
+ done
+}
+
+function unzip_from_source() {
+ tar -zxvf hadoop-"${HADOOP_VERSION}"-src.tar.gz
+ cd hadoop-"${HADOOP_VERSION}"-src
+}
+
+function rat_test() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_rat_test
+ mvn clean apache-rat:check "${MVN_PROPERTIES[@]}" 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_rat_test && RAT_CHECK_PASSED=1
+}
+
+function build_from_source() {
+ rm -f "${OUTPUT_PATH_PREFIX}"_build_from_source
+ # No unit test run.
+ mvn clean install "${MVN_PROPERTIES[@]}" -DskipTests 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_build_from_source && BUILD_FROM_SOURCE_PASSED=1
Review comment:
Sounds good, added an option to add list of profiles to use. Hence, now the command can be:
`./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC20/ -P native -P yarn-ui`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org