You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ko...@apache.org on 2021/12/13 04:10:26 UTC
[arrow] branch master updated: ARROW-14446: [Docs][Release] Update documentation on verification of release candidates
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new aa0e9d4 ARROW-14446: [Docs][Release] Update documentation on verification of release candidates
aa0e9d4 is described below
commit aa0e9d436c52a87fd1a99465e922ebacee072136
Author: Benson Muite <be...@emailplus.org>
AuthorDate: Mon Dec 13 13:08:20 2021 +0900
ARROW-14446: [Docs][Release] Update documentation on verification of release candidates
Make it easier for community verification
Closes #11562 from bkmgit/ARROW-14446
Lead-authored-by: Benson Muite <be...@emailplus.org>
Co-authored-by: Sutou Kouhei <ko...@clear-code.com>
Co-authored-by: bkmgit <be...@emailplus.org>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
.env | 1 +
dev/README.md | 7 ++--
dev/release/README.md | 2 ++
dev/release/VERIFY.md | 43 ++++++++++++++++++++++---
dev/release/setup-rhel-rebuilds.sh | 57 +++++++++++++++++++++++++++++++++
dev/release/setup-ubuntu.sh | 38 ++++++++++++++++++++++
dev/release/verify-release-candidate.sh | 13 ++++++--
dev/tasks/docker-tests/github.linux.yml | 12 +++++--
dev/tasks/tasks.yml | 28 +++++++++++++---
docker-compose.yml | 52 ++++++++++++++++++++++++------
10 files changed, 229 insertions(+), 24 deletions(-)
diff --git a/.env b/.env
index 7ebb92f..17fed07 100644
--- a/.env
+++ b/.env
@@ -46,6 +46,7 @@ REPO=apache/arrow-dev
ULIMIT_CORE=-1
# Default versions for platforms
+ALMALINUX=8
DEBIAN=11
FEDORA=33
UBUNTU=20.04
diff --git a/dev/README.md b/dev/README.md
index 16784b0..d861944 100644
--- a/dev/README.md
+++ b/dev/README.md
@@ -119,14 +119,15 @@ For now just say n as we have 1 branch
## Verifying Release Candidates
-We have provided a script to assist with verifying release candidates:
+We have provided a script to assist with verifying release candidates on Linux
+and macOS:
```shell
bash dev/release/verify-release-candidate.sh 0.7.0 0
```
-Currently this only works on Linux (patches to expand to macOS welcome!). Read
-the script for information about system dependencies.
+Read the script and check the notes in dev/release for information about system
+dependencies.
On Windows, we have a script that verifies C++ and Python (requires Visual
Studio 2015):
diff --git a/dev/release/README.md b/dev/release/README.md
index 0a9cc3e..e1ecdd4 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -22,3 +22,5 @@
To learn more, see the project wiki:
https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
+
+and [VERIFY.md](VERIFY.md)
diff --git a/dev/release/VERIFY.md b/dev/release/VERIFY.md
index 5b441ac..411e449 100644
--- a/dev/release/VERIFY.md
+++ b/dev/release/VERIFY.md
@@ -26,9 +26,9 @@ Windows. Read the comments in `verify-release-candidate.bat` for instructions.
## Linux and macOS
-We've provided a convenience script for verifying the C++, Python, C
-GLib, Java and JavaScript builds on Linux and macOS. Read the comments in
-`verify-release-candidate.sh` for instructions.
+We've provided a convenience script for verifying the C++, C#, C GLib, Go,
+Java, JavaScript, Ruby and Python builds on Linux and macOS. Read the script
+`verify-release-candidate.sh` for further information.
### C GLib
@@ -47,7 +47,7 @@ You can install them by the followings on Debian GNU/Linux and Ubuntu:
% sudo gem install gobject-introspection test-unit
```
-You can install them by the followings on CentOS:
+You can install them by the followings on CentOS 7:
```console
% sudo yum install -y gobject-introspection-devel
@@ -74,3 +74,38 @@ You need to set `PKG_CONFIG_PATH` to find libffi on macOS:
```console
% export PKG_CONFIG_PATH=$(brew --prefix libffi)/lib/pkgconfig:$PKG_CONFIG_PATH
```
+
+### C++, C#, C GLib, Go, Java, JavaScript, Python, Ruby
+
+Example scripts to install the dependencies to run the verification
+script for verifying the source on Ubuntu 20.04, Rocky Linux 8 and
+AlmaLinux 8 are in this folder and named `setup-ubuntu.sh` and
+`setup-rhel-rebuilds.sh`. These can be adapted to different
+situations. Go and JavaScript are installed by the verification
+script in the testing environment. Verifying the apt and yum binaries
+additionally requires installation of Docker.
+
+When verifying the source, by default the verification script will try
+to verify all implementations and bindings. Should one of the
+verification tests fail, the script will exit before running the other
+tests. It can be helpful to repeat the failed test to see if it will
+complete, since failures can occur for problems such as slow or failed
+download of a dependency from the internet. It is possible to run
+specific verification tests by setting environment variables, for example
+
+```console
+% TEST_DEFAULT=0 TEST_GO=1 dev/release/verify-release-candidate.sh source 6.0.0 3
+% TEST_YUM=1 dev/release/verify-release-candidate.sh binaries 6.0.0 3
+```
+
+It is also possible to use
+[Archery](https://arrow.apache.org/docs/developers/archery.html) to run
+the verification process in a container, for example
+
+```console
+% archery docker run -e VERIFY_VERSION=6.0.1 -e VERIFY_RC=1 almalinux-verify-rc-source
+% archery docker run -e VERIFY_VERSION=6.0.1 -e VERIFY_RC=1 ubuntu-verify-rc-source
+```
+
+To improve software quality, you are encouraged to verify
+on a variety of platforms.
diff --git a/dev/release/setup-rhel-rebuilds.sh b/dev/release/setup-rhel-rebuilds.sh
new file mode 100755
index 0000000..28ba78a
--- /dev/null
+++ b/dev/release/setup-rhel-rebuilds.sh
@@ -0,0 +1,57 @@
+#!/bin/sh
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# A script to install dependencies required for release
+# verification Red Hat Enterprise Linux 8 clones in particular
+# on AlmaLinux 8 and Rocky Linux 8
+
+dnf -y install \
+ 'dnf-command(config-manager)'
+dnf config-manager --set-enabled powertools
+dnf -y update
+dnf -y module disable nodejs
+dnf -y module enable nodejs:16
+dnf -y module disable ruby
+dnf -y module enable ruby:2.7
+dnf -y groupinstall "Development Tools"
+dnf -y install \
+ cmake \
+ git \
+ gobject-introspection-devel \
+ java-1.8.0-openjdk-devel \
+ libcurl-devel \
+ libcurl-devel \
+ llvm-devel \
+ llvm-toolset \
+ maven \
+ ncurses-devel \
+ ncurses-devel \
+ ninja-build \
+ ninja-build \
+ nodejs \
+ openssl-devel \
+ python3-devel \
+ python3-devel \
+ python3-pip \
+ python3-pip \
+ ruby-devel \
+ wget \
+ which
+npm install -g yarn
+alternatives --set python /usr/bin/python3
diff --git a/dev/release/setup-ubuntu.sh b/dev/release/setup-ubuntu.sh
new file mode 100755
index 0000000..f1327c0
--- /dev/null
+++ b/dev/release/setup-ubuntu.sh
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# A script to install dependencies required for release
+# verification on Ubuntu 20.04
+
+apt-get update
+apt-get -y install \
+ build-essential \
+ clang \
+ cmake \
+ curl \
+ git \
+ libgirepository1.0-dev \
+ libglib2.0-dev \
+ libssl-dev \
+ llvm-dev \
+ maven \
+ ninja-build \
+ openjdk-11-jdk \
+ pkg-config \
+ python3-pip \
+ ruby-dev \
+ wget
diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index 7964767..9ba06ff 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -25,6 +25,7 @@
# - gcc >= 4.8
# - Node.js >= 11.12 (best way is to use nvm)
# - Go >= 1.15
+# - Docker
#
# If using a non-system Boost, set BOOST_ROOT and add Boost libraries to
# LD_LIBRARY_PATH.
@@ -468,7 +469,7 @@ test_ruby() {
}
test_go() {
- local VERSION=1.15.14
+ local VERSION=1.16.12
local ARCH="$(uname -m)"
if [ "$ARCH" == "x86_64" ]; then
@@ -759,7 +760,15 @@ test_jars() {
# Install NodeJS locally for running the JavaScript tests rather than using the
# system Node installation, which may be too old.
-: ${INSTALL_NODE:=1}
+node_major_version=$( \
+ node --version 2>&1 | \grep -o '^v[0-9]*' | sed -e 's/^v//g' || :)
+required_node_major_version=14
+if [ -n "${node_major_version}" -a \
+ "${node_major_version}" -ge ${required_node_major_version} ]; then
+ : ${INSTALL_NODE:=0}
+else
+ : ${INSTALL_NODE:=1}
+fi
case "${ARTIFACT}" in
source)
diff --git a/dev/tasks/docker-tests/github.linux.yml b/dev/tasks/docker-tests/github.linux.yml
index 1faf482..b3c9b55 100644
--- a/dev/tasks/docker-tests/github.linux.yml
+++ b/dev/tasks/docker-tests/github.linux.yml
@@ -21,7 +21,8 @@
jobs:
test:
- name: Docker Test
+ name: |
+ Docker Test {{ flags|default("") }} {{ image }} {{ command|default("") }}
runs-on: ubuntu-latest
{% if env is defined %}
env:
@@ -39,7 +40,14 @@ jobs:
- name: Execute Docker Build
shell: bash
- run: archery docker run -e SETUPTOOLS_SCM_PRETEND_VERSION="{{ arrow.no_rc_version }}" {{ flags|default("") }} {{ image }} {{ command|default("") }}
+ run: |
+ archery docker run \
+ -e SETUPTOOLS_SCM_PRETEND_VERSION="{{ arrow.no_rc_version }}" \
+ -e VERIFY_VERSION="{{ release|default("") }}" \
+ -e VERIFY_RC="{{ rc|default("") }}" \
+ {{ flags|default("") }} \
+ {{ image }} \
+ {{ command|default("") }}
{% if '-r' in image %}
- name: Dump R install logs
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index e487fce..44fb2dc 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -843,8 +843,7 @@ tasks:
TEST_JARS: 1
artifact: jars
-{% for platform, arch, runner in [("linux", "amd64", "ubuntu-20.04"),
- ("macos", "amd64", "macos-10.15")] %}
+{% for platform, arch, runner in [("macos", "amd64", "macos-10.15")] %}
{% for target in ["cpp",
"csharp",
"go",
@@ -859,7 +858,6 @@ tasks:
template: verify-rc/github.{{ platform }}.{{ arch }}.yml
params:
env:
- INSTALL_NODE: 0
TEST_DEFAULT: 0
TEST_{{ target|upper }}: 1
artifact: "source"
@@ -867,6 +865,29 @@ tasks:
{% endfor %}
{% endfor %}
+{% for distribution, version in [("almalinux", "8"),
+ ("ubuntu", "18.04"),
+ ("ubuntu", "20.04")] %}
+ {% for target in ["cpp",
+ "csharp",
+ "go",
+ "integration",
+ "java",
+ "js",
+ "python",
+ "ruby"] %}
+
+ verify-rc-source-{{ target }}-linux-{{ distribution }}-{{ version }}-amd64:
+ ci: github
+ template: docker-tests/github.linux.yml
+ params:
+ flags: >-
+ -e TEST_DEFAULT=0
+ -e TEST_{{ target|upper }}=1
+ image: {{ distribution }}-verify-rc-source
+ {% endfor %}
+{% endfor %}
+
{% for platform, arch, runner in [("macos", "arm64", "self-hosted")] %}
{% for target in ["cpp",
"csharp",
@@ -883,7 +904,6 @@ tasks:
env:
ARROW_FLIGHT: 0
ARROW_GANDIVA: 0
- INSTALL_NODE: 0
TEST_DEFAULT: 0
TEST_INTEGRATION_JAVA: 0
TEST_{{ target|upper }}: 1
diff --git a/docker-compose.yml b/docker-compose.yml
index 7f96c2c..834304d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -95,6 +95,7 @@ x-hierarchy:
# Each node must be either a string scalar of a list containing the
# descendant images if any. Archery checks that all node has a corresponding
# service entry, so any new image/service must be listed here.
+ - almalinux-verify-rc-source
- conda:
- conda-cpp:
- conda-integration
@@ -105,11 +106,11 @@ x-hierarchy:
- conda-python-docs
- conda-python-dask
- conda-python-hdfs
- - conda-python-jpype
- conda-python-java-integration
- - conda-python-turbodbc
+ - conda-python-jpype
- conda-python-kartothek
- conda-python-spark
+ - conda-python-turbodbc
- debian-cpp:
- debian-c-glib:
- debian-ruby
@@ -122,6 +123,7 @@ x-hierarchy:
- debian-js
- fedora-cpp:
- fedora-python
+ - python-sdist
- ubuntu-cpp:
- ubuntu-c-glib:
- ubuntu-ruby
@@ -138,10 +140,10 @@ x-hierarchy:
- ubuntu-cpp-sanitizer
- ubuntu-cpp-thread-sanitizer
- ubuntu-r-sanitizer
+ - ubuntu-r-valgrind
+ - ubuntu-verify-rc-source
- r-fedora-clang-devel-san:
- fedora-r-clang-sanitizer
- - ubuntu-r-valgrind
- - python-sdist
- r
- r-revdepcheck
# helper services
@@ -160,12 +162,10 @@ volumes:
name: ${ARCH}-conda-ccache
debian-ccache:
name: ${ARCH}-debian-${DEBIAN}-ccache
- ubuntu-ccache:
- name: ${ARCH}-ubuntu-${UBUNTU}-ccache
- fedora-ccache:
- name: ${ARCH}-fedora-${FEDORA}-ccache
debian-rust:
name: ${ARCH}-debian-${DEBIAN}-rust
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
maven-cache:
name: maven-cache
python-wheel-manylinux2010-ccache:
@@ -174,7 +174,9 @@ volumes:
name: python-wheel-manylinux2014-ccache
python-wheel-windows-clcache:
name: python-wheel-windows-clcache
-
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+
services:
################################# C++ #######################################
@@ -1696,3 +1698,35 @@ services:
/arrow/ci/scripts/python_build.sh /arrow /build &&
/arrow/ci/scripts/java_build.sh /arrow /build &&
/arrow/ci/scripts/integration_spark.sh /arrow /spark ${TEST_PYARROW_ONLY:-false}"]
+
+ ################################# Source Verification #####################################
+
+ almalinux-verify-rc-source:
+ # Usage:
+ # docker-compose build almalinux-verify-rc-source
+ # docker-compose run -e VERIFY_VERSION=6.0.1 -e VERIFY_RC=1 almalinux-verify-rc-source
+ # Parameters:
+ # ALMALINUX: 8
+ image: almalinux:${ALMALINUX}
+ volumes:
+ - .:/arrow:delegated
+ shm_size: '1gb'
+ command: >
+ /bin/bash -c "
+ /arrow/dev/release/setup-rhel-rebuilds.sh &&
+ /arrow/dev/release/verify-release-candidate.sh source $${VERIFY_VERSION} $${VERIFY_RC}"
+
+ ubuntu-verify-rc-source:
+ # Usage:
+ # docker-compose build ubuntu-verify-rc-source
+ # docker-compose run -e VERIFY_VERSION=6.0.1 -e VERIFY_RC=1 ubuntu-verify-rc-source
+ # Parameters:
+ # UBUNTU: 18.04, 20.04
+ image: ubuntu:${UBUNTU}
+ volumes:
+ - .:/arrow:delegated
+ shm_size: '1gb'
+ command: >
+ /bin/bash -c "
+ DEBIAN_FRONTEND=noninteractive /arrow/dev/release/setup-ubuntu.sh &&
+ /arrow/dev/release/verify-release-candidate.sh source $${VERIFY_VERSION} $${VERIFY_RC}"