You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by sl...@apache.org on 2025/12/08 23:10:44 UTC
(hadoop) branch branch-3.4 updated (224a102b66d -> 721bdabe3cb)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a change to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
from 224a102b66d HADOOP-19745. License header verification failure in hadoop-assemblies (#8112) (#8115)
new b793e111d82 HADOOP-19495. Add JDK 24 to Ubuntu 20.04 docker development images.
new 2a6df7ecc19 HADOOP-19540. [JDK17] Add ubuntu:noble as a build platform with JDK-17 as default (#7608)
new a28f7309c9c HADOOP-19570. Upgrade libxxhash to 0.8.3 in Windows 10 (#7689)
new ffdb112acfe HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. (#7822)
new 48bdbb75be8 HADOOP-19661. Migrate CentOS 8 to Rocky Linux 8 in build env Dockerfile
new a5c8631d0cc HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8 (#7917)
new e50ab07c11b HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile (#7898) Contributed by Cheng Pan.
new 0d00b7885f9 HADOOP-19661. (Follow-up) Install missing deps for Rocky Linux 8 docker image (#7931) Contributed by Cheng Pan.
new 721bdabe3cb HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image (#7932) Contributed by Cheng Pan.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
BUILDING.txt | 10 +-
dev-support/Jenkinsfile | 84 ++------
dev-support/docker/Dockerfile | 5 +
dev-support/docker/Dockerfile_aarch64 | 5 +
dev-support/docker/Dockerfile_centos_7 | 108 ----------
.../{Dockerfile_debian_10 => Dockerfile_debian_11} | 40 ++--
...Dockerfile_centos_8 => Dockerfile_rockylinux_8} | 82 ++++----
.../docker/{Dockerfile => Dockerfile_ubuntu_24} | 22 +-
dev-support/docker/Dockerfile_windows_10 | 8 +-
dev-support/docker/pkg-resolver/install-epel.sh | 2 +-
.../docker/pkg-resolver/install-pkg-resolver.sh | 2 +-
dev-support/docker/pkg-resolver/packages.json | 233 +++++++++++----------
dev-support/docker/pkg-resolver/platforms.json | 8 +-
.../pkg-resolver/set-vault-as-baseurl-centos.sh | 33 ---
hadoop-project/pom.xml | 10 +
start-build-env.sh | 26 ++-
16 files changed, 279 insertions(+), 399 deletions(-)
delete mode 100644 dev-support/docker/Dockerfile_centos_7
rename dev-support/docker/{Dockerfile_debian_10 => Dockerfile_debian_11} (71%)
rename dev-support/docker/{Dockerfile_centos_8 => Dockerfile_rockylinux_8} (55%)
copy dev-support/docker/{Dockerfile => Dockerfile_ubuntu_24} (76%)
delete mode 100644 dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 05/09: HADOOP-19661. Migrate CentOS 8 to Rocky Linux 8 in build env Dockerfile
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 48bdbb75be81065810dd87111dd95bafa1777f3b
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Fri Aug 29 05:56:46 2025 +0000
HADOOP-19661. Migrate CentOS 8 to Rocky Linux 8 in build env Dockerfile
Closes #7900
Signed-off-by: Shilun Fan <sl...@apache.org>
Signed-off-by: Chris Nauroth <cn...@apache.org>
---
BUILDING.txt | 6 +-
dev-support/Jenkinsfile | 14 ++--
dev-support/docker/Dockerfile_centos_8 | 82 ++++++++++++----------
dev-support/docker/pkg-resolver/install-epel.sh | 2 +-
dev-support/docker/pkg-resolver/packages.json | 44 ++++++------
dev-support/docker/pkg-resolver/platforms.json | 4 +-
.../pkg-resolver/set-vault-as-baseurl-centos.sh | 33 ---------
start-build-env.sh | 2 +-
8 files changed, 81 insertions(+), 106 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index 8795880c552..9a7af891db6 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -31,7 +31,7 @@ On Linux / Mac:
$ ./start-build-env.sh [OS platform]
- - [OS Platform] One of [centos_7, centos_8, debian_10, ubuntu_20, ubuntu_24, windows_10].
+ - [OS Platform] One of [rockylinux_8, debian_10, ubuntu_20, ubuntu_24, windows_10].
Default is 'ubuntu_20'.
Note: Currently only default ('ubuntu_20') is supported on arm machine
@@ -505,7 +505,7 @@ path. This is necessary at least for Homebrewed OpenSSL.
----------------------------------------------------------------------------------
-Building on CentOS 8
+Building on Rocky Linux 8
----------------------------------------------------------------------------------
@@ -526,7 +526,7 @@ Building on CentOS 8
$ sudo make install
$ cd ..
-* Install libraries provided by CentOS 8.
+* Install libraries provided by Rocky Linux 8.
$ sudo dnf install libtirpc-devel zlib-devel lz4-devel bzip2-devel openssl-devel cyrus-sasl-devel libpmem-devel
* Install GCC 9.3.0
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 06095cc8a4f..9ed78f1ff32 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -84,10 +84,10 @@ pipeline {
stage ('setup sources') {
steps {
- dir("${WORKSPACE}/centos-8") {
+ dir("${WORKSPACE}/rockylinux-8") {
sh '''#!/usr/bin/env bash
- cp -Rp ${WORKSPACE}/src ${WORKSPACE}/centos-8
+ cp -Rp ${WORKSPACE}/src ${WORKSPACE}/rockylinux-8
'''
}
@@ -111,11 +111,11 @@ pipeline {
// C++/C++ build/platform.
// This stage serves as a means of cross platform validation, which is
// really needed to ensure that any C++ related/platform change doesn't
- // break the Hadoop build on Centos 8.
- stage ('precommit-run Centos 8') {
+ // break the Hadoop build on Rocky Linux 8.
+ stage ('precommit-run Rocky Linux 8') {
environment {
- SOURCEDIR = "${WORKSPACE}/centos-8/src"
- PATCHDIR = "${WORKSPACE}/centos-8/out"
+ SOURCEDIR = "${WORKSPACE}/rockylinux-8/src"
+ PATCHDIR = "${WORKSPACE}/rockylinux-8/out"
DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_centos_8"
IS_OPTIONAL = 1
}
@@ -136,7 +136,7 @@ pipeline {
failure {
sh '''#!/usr/bin/env bash
- cp -Rp "${WORKSPACE}/centos-8/out" "${WORKSPACE}"
+ cp -Rp "${WORKSPACE}/rockylinux-8/out" "${WORKSPACE}"
'''
archiveArtifacts "out/**"
}
diff --git a/dev-support/docker/Dockerfile_centos_8 b/dev-support/docker/Dockerfile_centos_8
index ee0c8e88f74..63c22c5e3e1 100644
--- a/dev-support/docker/Dockerfile_centos_8
+++ b/dev-support/docker/Dockerfile_centos_8
@@ -17,7 +17,7 @@
# Dockerfile for installing the necessary dependencies for building Hadoop.
# See BUILDING.txt.
-FROM centos:8
+FROM rockylinux:8
WORKDIR /root
@@ -30,25 +30,18 @@ COPY pkg-resolver pkg-resolver
RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
&& chmod a+r pkg-resolver/*.json
-######
-# Centos 8 has reached its EOL and the packages
-# are no longer available on mirror.centos.org site.
-# Please see https://www.centos.org/centos-linux-eol/
-######
-RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:8
-
######
# Install packages from yum
######
# hadolint ignore=DL3008,SC2046
RUN yum update -y \
&& yum install -y python3 \
- && yum install -y $(pkg-resolver/resolve.py centos:8)
+ && yum install -y $(pkg-resolver/resolve.py rockylinux:8)
####
# Install EPEL
####
-RUN pkg-resolver/install-epel.sh centos:8
+RUN pkg-resolver/install-epel.sh rockylinux:8
RUN dnf --enablerepo=powertools install -y \
doxygen \
@@ -69,50 +62,65 @@ SHELL ["/bin/bash", "--login", "-c"]
# Set the environment variables needed for CMake
# to find and use GCC 9 for compilation
######
-ENV GCC_HOME "/opt/rh/gcc-toolset-9"
-ENV CC "${GCC_HOME}/root/usr/bin/gcc"
-ENV CXX "${GCC_HOME}/root/usr/bin/g++"
-ENV MODULES_RUN_QUARANTINE "LD_LIBRARY_PATH LD_PRELOAD"
-ENV MODULES_CMD "/usr/share/Modules/libexec/modulecmd.tcl"
-ENV SHLVL 1
-ENV MODULEPATH "/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles"
-ENV MODULEPATH_modshare "/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1"
-ENV MODULESHOME "/usr/share/Modules"
-ENV LD_LIBRARY_PATH "${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:${GCC_HOME}/root/usr/lib64/dyninst:${GCC_HOME}/root/usr/lib/dyninst:${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:/usr/lib:/usr/lib64"
-ENV PCP_DIR "${GCC_HOME}/root"
-ENV MANPATH "${GCC_HOME}/root/usr/share/man::"
-ENV PATH "${GCC_HOME}/root/usr/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
-ENV PKG_CONFIG_PATH "${GCC_HOME}/root/usr/lib64/pkgconfig"
-ENV INFOPATH "${GCC_HOME}/root/usr/share/info"
+ENV GCC_HOME="/opt/rh/gcc-toolset-9"
+ENV CC="${GCC_HOME}/root/usr/bin/gcc"
+ENV CXX="${GCC_HOME}/root/usr/bin/g++"
+ENV MODULES_RUN_QUARANTINE="LD_LIBRARY_PATH LD_PRELOAD"
+ENV MODULES_CMD="/usr/share/Modules/libexec/modulecmd.tcl"
+ENV SHLVL=1
+ENV MODULEPATH="/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles"
+ENV MODULEPATH_modshare="/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1"
+ENV MODULESHOME="/usr/share/Modules"
+ENV LD_LIBRARY_PATH="${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:${GCC_HOME}/root/usr/lib64/dyninst:${GCC_HOME}/root/usr/lib/dyninst:${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:/usr/lib:/usr/lib64"
+ENV PCP_DIR="${GCC_HOME}/root"
+ENV MANPATH="${GCC_HOME}/root/usr/share/man::"
+ENV PATH="${GCC_HOME}/root/usr/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+ENV PKG_CONFIG_PATH="${GCC_HOME}/root/usr/lib64/pkgconfig"
+ENV INFOPATH="${GCC_HOME}/root/usr/share/info"
# TODO: Set locale
######
# Set env vars required to build Hadoop
######
-ENV MAVEN_HOME /opt/maven
-ENV PATH "${PATH}:${MAVEN_HOME}/bin"
+ENV MAVEN_HOME=/opt/maven
+ENV PATH="${PATH}:${MAVEN_HOME}/bin"
# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
-ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
+ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0
#######
# Set env vars for SpotBugs
#######
-ENV SPOTBUGS_HOME /opt/spotbugs
+ENV SPOTBUGS_HOME=/opt/spotbugs
#######
# Set env vars for Google Protobuf 3.21.12
#######
-ENV PROTOBUF_HOME /opt/protobuf
-ENV PATH "${PATH}:/opt/protobuf/bin"
+ENV PROTOBUF_HOME=/opt/protobuf
+ENV PATH="${PATH}:${PROTOBUF_HOME}/bin"
+
+# Skip gpg verification when downloading Yetus via yetus-wrapper
+ENV HADOOP_SKIP_YETUS_VERIFICATION=true
######
# Install packages
######
-RUN pkg-resolver/install-maven.sh centos:8
-RUN pkg-resolver/install-cmake.sh centos:8
-RUN pkg-resolver/install-boost.sh centos:8
-RUN pkg-resolver/install-spotbugs.sh centos:8
-RUN pkg-resolver/install-protobuf.sh centos:8
-RUN pkg-resolver/install-zstandard.sh centos:8
+RUN pkg-resolver/install-maven.sh rockylinux:8
+RUN pkg-resolver/install-cmake.sh rockylinux:8
+RUN pkg-resolver/install-boost.sh rockylinux:8
+RUN pkg-resolver/install-spotbugs.sh rockylinux:8
+RUN pkg-resolver/install-protobuf.sh rockylinux:8
+RUN pkg-resolver/install-zstandard.sh rockylinux:8
RUN pkg-resolver/install-common-pkgs.sh
+
+###
+# Everything past this point is either not needed for testing or breaks Yetus.
+# So tell Yetus not to read the rest of the file:
+# YETUS CUT HERE
+###
+
+# Add a welcome message and environment checks.
+COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh
+RUN chmod 755 /root/hadoop_env_checks.sh
+# hadolint ignore=SC2016
+RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc
diff --git a/dev-support/docker/pkg-resolver/install-epel.sh b/dev-support/docker/pkg-resolver/install-epel.sh
index 875dce3a9ae..b6eabffca02 100644
--- a/dev-support/docker/pkg-resolver/install-epel.sh
+++ b/dev-support/docker/pkg-resolver/install-epel.sh
@@ -40,7 +40,7 @@ fi
if [ "$version_to_install" == "8" ]; then
mkdir -p /tmp/epel &&
- curl -L -s -S https://download-ib01.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm \
+ curl -L -s -S https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm \
-o /tmp/epel/epel-release-latest-8.noarch.rpm &&
rpm -Uvh /tmp/epel/epel-release-latest-8.noarch.rpm
else
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index efa6b1726ce..28b8733e04b 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -4,7 +4,7 @@
"ubuntu:focal": "ant",
"ubuntu:focal::arch64": "ant",
"ubuntu:noble": "ant",
- "centos:8": "ant"
+ "rockylinux:8": "ant"
},
"apt-utils": {
"debian:10": "apt-utils",
@@ -17,7 +17,7 @@
"ubuntu:focal": "automake",
"ubuntu:noble": "automake",
"ubuntu:focal::arch64": "automake",
- "centos:8": "automake"
+ "rockylinux:8": "automake"
},
"bats": {
"debian:10": "bats",
@@ -48,7 +48,7 @@
"bzip2",
"libbz2-dev"
],
- "centos:8": [
+ "rockylinux:8": [
"bzip2",
"bzip2-devel"
]
@@ -58,7 +58,7 @@
"ubuntu:focal": "clang",
"ubuntu:noble": "clang",
"ubuntu:focal::arch64": "clang",
- "centos:8": "clang"
+ "rockylinux:8": "clang"
},
"cmake": {
"ubuntu:focal": "cmake",
@@ -82,7 +82,7 @@
"curl",
"libcurl4-openssl-dev"
],
- "centos:8": [
+ "rockylinux:8": [
"curl",
"libcurl-devel"
]
@@ -94,7 +94,7 @@
"ubuntu:focal::arch64": "doxygen"
},
"dnf": {
- "centos:8": "dnf"
+ "rockylinux:8": "dnf"
},
"fuse": {
"debian:10": [
@@ -113,7 +113,7 @@
"fuse",
"libfuse-dev"
],
- "centos:8": [
+ "rockylinux:8": [
"fuse",
"fuse-libs",
"fuse-devel"
@@ -144,7 +144,7 @@
"ubuntu:focal": "git",
"ubuntu:noble": "git",
"ubuntu:focal::arch64": "git",
- "centos:8": "git"
+ "rockylinux:8": "git"
},
"gnupg-agent": {
"debian:10": "gnupg-agent",
@@ -169,17 +169,17 @@
"ubuntu:focal": "libtool",
"ubuntu:noble": "libtool",
"ubuntu:focal::arch64": "libtool",
- "centos:8": "libtool"
+ "rockylinux:8": "libtool"
},
"openssl": {
"debian:10": "libssl-dev",
"ubuntu:focal": "libssl-dev",
"ubuntu:noble": "libssl-dev",
"ubuntu:focal::arch64": "libssl-dev",
- "centos:8": "openssl-devel"
+ "rockylinux:8": "openssl-devel"
},
"perl": {
- "centos:7": [
+ "rockylinux:7": [
"perl-CPAN",
"perl-devel"
]
@@ -207,7 +207,7 @@
"ubuntu:focal": "libsasl2-dev",
"ubuntu:noble": "libsasl2-dev",
"ubuntu:focal::arch64": "libsasl2-dev",
- "centos:8": "cyrus-sasl-devel"
+ "rockylinux:8": "cyrus-sasl-devel"
},
"snappy": {
"debian:10": "libsnappy-dev",
@@ -232,7 +232,7 @@
"libzstd-dev",
"zlib1g-dev"
],
- "centos:8": [
+ "rockylinux:8": [
"zlib-devel",
"lz4-devel"
]
@@ -244,7 +244,7 @@
"ubuntu:focal::arch64": "locales"
},
"libtirpc-devel": {
- "centos:8": "libtirpc-devel",
+ "rockylinux:8": "libtirpc-devel",
"ubuntu:noble": "libtirpc-dev"
},
"make": {
@@ -252,7 +252,7 @@
"ubuntu:focal": "make",
"ubuntu:noble": "make",
"ubuntu:focal::arch64": "make",
- "centos:8": "make"
+ "rockylinux:8": "make"
},
"maven": {
"debian:10": "maven",
@@ -285,14 +285,14 @@
"ubuntu:focal": "pinentry-curses",
"ubuntu:noble": "pinentry-curses",
"ubuntu:focal::arch64": "pinentry-curses",
- "centos:8": "pinentry-curses"
+ "rockylinux:8": "pinentry-curses"
},
"pkg-config": {
"debian:10": "pkg-config",
"ubuntu:focal": "pkg-config",
"ubuntu:noble": "pkg-config",
"ubuntu:focal::arch64": "pkg-config",
- "centos:8": "pkg-config"
+ "rockylinux:8": "pkg-config"
},
"python": {
"debian:10": [
@@ -324,7 +324,7 @@
"python3-setuptools",
"python3-wheel"
],
- "centos:8": [
+ "rockylinux:8": [
"python3",
"python3-pip",
"python3-setuptools",
@@ -336,7 +336,7 @@
"ubuntu:focal": "rsync",
"ubuntu:noble": "rsync",
"ubuntu:focal::arch64": "rsync",
- "centos:8": "rsync"
+ "rockylinux:8": "rsync"
},
"shellcheck": {
"debian:10": "shellcheck",
@@ -345,7 +345,7 @@
"ubuntu:focal::arch64": "shellcheck"
},
"shasum": {
- "centos:8": "perl-Digest-SHA"
+ "rockylinux:8": "perl-Digest-SHA"
},
"software-properties-common": {
"debian:10": "software-properties-common",
@@ -358,14 +358,14 @@
"ubuntu:focal": "sudo",
"ubuntu:noble": "sudo",
"ubuntu:focal::arch64": "sudo",
- "centos:8": "sudo"
+ "rockylinux:8": "sudo"
},
"valgrind": {
"debian:10": "valgrind",
"ubuntu:focal": "valgrind",
"ubuntu:noble": "valgrind",
"ubuntu:focal::arch64": "valgrind",
- "centos:8": "valgrind"
+ "rockylinux:8": "valgrind"
},
"yasm": {
"debian:10": "yasm",
diff --git a/dev-support/docker/pkg-resolver/platforms.json b/dev-support/docker/pkg-resolver/platforms.json
index 47842cac9b4..eebf7fc948f 100644
--- a/dev-support/docker/pkg-resolver/platforms.json
+++ b/dev-support/docker/pkg-resolver/platforms.json
@@ -2,6 +2,6 @@
"ubuntu:focal",
"ubuntu:focal::arch64",
"ubuntu:noble",
- "centos:8",
+ "rockylinux:8",
"debian:10"
-]
\ No newline at end of file
+]
diff --git a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh b/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh
deleted file mode 100644
index 905ac5077de..00000000000
--- a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/usr/bin/env bash
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-if [ $# -lt 1 ]; then
- echo "ERROR: Need at least 1 argument, $# were provided"
- exit 1
-fi
-
-if [ "$1" == "centos:7" ] || [ "$1" == "centos:8" ]; then
- cd /etc/yum.repos.d/ || exit &&
- sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* &&
- sed -i 's|# *baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* &&
- yum update -y &&
- cd /root || exit
-else
- echo "ERROR: Setting the archived baseurl is only supported for centos 7 and 8 environments"
- exit 1
-fi
diff --git a/start-build-env.sh b/start-build-env.sh
index de82d7e64ab..b97eca8ad74 100755
--- a/start-build-env.sh
+++ b/start-build-env.sh
@@ -92,7 +92,7 @@ RUN rm -f /var/log/faillog /var/log/lastlog
RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME} -d "${DOCKER_HOME_DIR}"
RUN echo "${USER_NAME} ALL=NOPASSWD: ALL" > "/etc/sudoers.d/hadoop-build-${USER_ID}"
-ENV HOME "${DOCKER_HOME_DIR}"
+ENV HOME="${DOCKER_HOME_DIR}"
UserSpecificDocker
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 03/09: HADOOP-19570. Upgrade libxxhash to 0.8.3 in Windows 10 (#7689)
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a28f7309c9cbfe4bd5bde0159a8445ad1dd113e7
Author: Gautham B A <ga...@gmail.com>
AuthorDate: Fri May 16 21:25:41 2025 +0530
HADOOP-19570. Upgrade libxxhash to 0.8.3 in Windows 10 (#7689)
* The current version of libxxhash - 0.8.1 isn't
available on the msys repo.
* This is causing the Hadoop Jenkins CI for
Windows to fail while building the docker image
for Windows 10.
* This PR upgrades libxxhash to 0.8.3 to fix this
issue.
---
dev-support/docker/Dockerfile_windows_10 | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/dev-support/docker/Dockerfile_windows_10 b/dev-support/docker/Dockerfile_windows_10
index 105529c5d65..7d32e6a84ff 100644
--- a/dev-support/docker/Dockerfile_windows_10
+++ b/dev-support/docker/Dockerfile_windows_10
@@ -79,11 +79,11 @@ RUN powershell zstd -d $Env:TEMP\libopenssl-3.1.0-1-x86_64.pkg.tar.zst -o $Env:T
RUN powershell mkdir "C:\LibOpenSSL"
RUN powershell tar -xvf $Env:TEMP\libopenssl-3.1.0-1-x86_64.pkg.tar -C "C:\LibOpenSSL"
-# Install libxxhash 0.8.1 needed for rsync 3.2.7.
-RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libxxhash-0.8.1-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libxxhash-0.8.1-1-x86_64.pkg.tar.zst
-RUN powershell zstd -d $Env:TEMP\libxxhash-0.8.1-1-x86_64.pkg.tar.zst -o $Env:TEMP\libxxhash-0.8.1-1-x86_64.pkg.tar
+# Install libxxhash 0.8.3 needed for rsync 3.2.7.
+RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libxxhash-0.8.3-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst
+RUN powershell zstd -d $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar.zst -o $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar
RUN powershell mkdir "C:\LibXXHash"
-RUN powershell tar -xvf $Env:TEMP\libxxhash-0.8.1-1-x86_64.pkg.tar -C "C:\LibXXHash"
+RUN powershell tar -xvf $Env:TEMP\libxxhash-0.8.3-1-x86_64.pkg.tar -C "C:\LibXXHash"
# Install libzstd 1.5.4 needed for rsync 3.2.7.
RUN powershell Invoke-WebRequest -Uri https://repo.msys2.org/msys/x86_64/libzstd-1.5.4-1-x86_64.pkg.tar.zst -OutFile $Env:TEMP\libzstd-1.5.4-1-x86_64.pkg.tar.zst
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 07/09: HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile (#7898) Contributed by Cheng Pan.
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit e50ab07c11be2bad830f531eb2e0646fbca82253
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Mon Sep 1 23:31:41 2025 +0800
HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile (#7898) Contributed by Cheng Pan.
* HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile
Signed-off-by: Shilun Fan <sl...@apache.org>
---
BUILDING.txt | 2 +-
dev-support/Jenkinsfile | 16 ++---
.../{Dockerfile_debian_10 => Dockerfile_debian_11} | 40 ++++++------
.../docker/pkg-resolver/install-pkg-resolver.sh | 2 +-
dev-support/docker/pkg-resolver/packages.json | 71 +++++++++++-----------
dev-support/docker/pkg-resolver/platforms.json | 2 +-
6 files changed, 67 insertions(+), 66 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index 9a7af891db6..73c50c1f67f 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -31,7 +31,7 @@ On Linux / Mac:
$ ./start-build-env.sh [OS platform]
- - [OS Platform] One of [rockylinux_8, debian_10, ubuntu_20, ubuntu_24, windows_10].
+ - [OS Platform] One of [rockylinux_8, debian_11, ubuntu_20, ubuntu_24, windows_10].
Default is 'ubuntu_20'.
Note: Currently only default ('ubuntu_20') is supported on arm machine
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 2a2b52b20f9..672061ea09a 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -91,10 +91,10 @@ pipeline {
'''
}
- dir("${WORKSPACE}/debian-10") {
+ dir("${WORKSPACE}/debian-11") {
sh '''#!/usr/bin/env bash
- cp -Rp ${WORKSPACE}/src ${WORKSPACE}/debian-10
+ cp -Rp ${WORKSPACE}/src ${WORKSPACE}/debian-11
'''
}
@@ -157,12 +157,12 @@ pipeline {
// C++/C++ build/platform.
// This stage serves as a means of cross platform validation, which is
// really needed to ensure that any C++ related/platform change doesn't
- // break the Hadoop build on Debian 10.
- stage ('precommit-run Debian 10') {
+ // break the Hadoop build on Debian 11.
+ stage ('precommit-run Debian 11') {
environment {
- SOURCEDIR = "${WORKSPACE}/debian-10/src"
- PATCHDIR = "${WORKSPACE}/debian-10/out"
- DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_debian_10"
+ SOURCEDIR = "${WORKSPACE}/debian-11/src"
+ PATCHDIR = "${WORKSPACE}/debian-11/out"
+ DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_debian_11"
IS_OPTIONAL = 1
}
@@ -182,7 +182,7 @@ pipeline {
failure {
sh '''#!/usr/bin/env bash
- cp -Rp "${WORKSPACE}/debian-10/out" "${WORKSPACE}"
+ cp -Rp "${WORKSPACE}/debian-11/out" "${WORKSPACE}"
'''
archiveArtifacts "out/**"
}
diff --git a/dev-support/docker/Dockerfile_debian_10 b/dev-support/docker/Dockerfile_debian_11
similarity index 71%
rename from dev-support/docker/Dockerfile_debian_10
rename to dev-support/docker/Dockerfile_debian_11
index 71446b27f68..2dc47eee6ac 100644
--- a/dev-support/docker/Dockerfile_debian_10
+++ b/dev-support/docker/Dockerfile_debian_11
@@ -17,7 +17,7 @@
# Dockerfile for installing the necessary dependencies for building Hadoop.
# See BUILDING.txt.
-FROM debian:10
+FROM debian:11
WORKDIR /root
@@ -29,26 +29,24 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
-ENV DEBIAN_FRONTEND noninteractive
-ENV DEBCONF_TERSE true
+ENV DEBIAN_FRONTEND=noninteractive
+ENV DEBCONF_TERSE=true
######
# Platform package dependency resolver
######
COPY pkg-resolver pkg-resolver
RUN chmod a+x pkg-resolver/install-pkg-resolver.sh
-RUN pkg-resolver/install-pkg-resolver.sh debian:10
+RUN pkg-resolver/install-pkg-resolver.sh debian:11
######
# Install packages from apt
######
# hadolint ignore=DL3008,SC2046
RUN apt-get -q update \
- && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:10) \
- && echo 'deb http://deb.debian.org/debian bullseye main' >> /etc/apt/sources.list \
- && apt-get -q update \
- && apt-get -q install -y --no-install-recommends -t bullseye $(pkg-resolver/resolve.py --release=bullseye debian:10) \
+ && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py debian:11) \
&& apt-get clean \
+ && update-java-alternatives -s java-1.11.0-openjdk-amd64 \
&& rm -rf /var/lib/apt/lists/*
# TODO : Set locale
@@ -56,38 +54,38 @@ RUN apt-get -q update \
######
# Set env vars required to build Hadoop
######
-ENV MAVEN_HOME /usr
+ENV MAVEN_HOME=/usr
# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
-ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
+ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
#######
# Set env vars for SpotBugs 4.2.2
#######
-ENV SPOTBUGS_HOME /opt/spotbugs
+ENV SPOTBUGS_HOME=/opt/spotbugs
#######
# Set env vars for Google Protobuf 3.21.12
#######
-ENV PROTOBUF_HOME /opt/protobuf
-ENV PATH "${PATH}:/opt/protobuf/bin"
+ENV PROTOBUF_HOME=/opt/protobuf
+ENV PATH="${PATH}:/opt/protobuf/bin"
###
# Avoid out of memory errors in builds
###
-ENV MAVEN_OPTS -Xms256m -Xmx3072m
+ENV MAVEN_OPTS="-Xms256m -Xmx3072m"
# Skip gpg verification when downloading Yetus via yetus-wrapper
-ENV HADOOP_SKIP_YETUS_VERIFICATION true
+ENV HADOOP_SKIP_YETUS_VERIFICATION=true
####
# Install packages
####
-RUN pkg-resolver/install-cmake.sh debian:10
-RUN pkg-resolver/install-spotbugs.sh debian:10
-RUN pkg-resolver/install-boost.sh debian:10
-RUN pkg-resolver/install-protobuf.sh debian:10
-RUN pkg-resolver/install-hadolint.sh debian:10
-RUN pkg-resolver/install-intel-isa-l.sh debian:10
+RUN pkg-resolver/install-cmake.sh debian:11
+RUN pkg-resolver/install-spotbugs.sh debian:11
+RUN pkg-resolver/install-boost.sh debian:11
+RUN pkg-resolver/install-protobuf.sh debian:11
+RUN pkg-resolver/install-hadolint.sh debian:11
+RUN pkg-resolver/install-intel-isa-l.sh debian:11
###
# Everything past this point is either not needed for testing or breaks Yetus.
diff --git a/dev-support/docker/pkg-resolver/install-pkg-resolver.sh b/dev-support/docker/pkg-resolver/install-pkg-resolver.sh
index 70e94b3792d..81dd0002f40 100644
--- a/dev-support/docker/pkg-resolver/install-pkg-resolver.sh
+++ b/dev-support/docker/pkg-resolver/install-pkg-resolver.sh
@@ -24,7 +24,7 @@ fi
chmod a+x pkg-resolver/*.sh pkg-resolver/*.py
chmod a+r pkg-resolver/*.json
-if [ "$1" == "debian:10" ]; then
+if [ "$1" == "debian:11" ]; then
apt-get -q update
apt-get -q install -y --no-install-recommends python3 \
python3-pip \
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 28b8733e04b..452ed015c32 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -1,38 +1,38 @@
{
"ant": {
- "debian:10": "ant",
+ "debian:11": "ant",
"ubuntu:focal": "ant",
"ubuntu:focal::arch64": "ant",
"ubuntu:noble": "ant",
"rockylinux:8": "ant"
},
"apt-utils": {
- "debian:10": "apt-utils",
+ "debian:11": "apt-utils",
"ubuntu:focal": "apt-utils",
"ubuntu:noble": "apt-utils",
"ubuntu:focal::arch64": "apt-utils"
},
"automake": {
- "debian:10": "automake",
+ "debian:11": "automake",
"ubuntu:focal": "automake",
"ubuntu:noble": "automake",
"ubuntu:focal::arch64": "automake",
"rockylinux:8": "automake"
},
"bats": {
- "debian:10": "bats",
+ "debian:11": "bats",
"ubuntu:focal": "bats",
"ubuntu:noble": "bats",
"ubuntu:focal::arch64": "bats"
},
"build-essential": {
- "debian:10": "build-essential",
+ "debian:11": "build-essential",
"ubuntu:focal": "build-essential",
"ubuntu:noble": "build-essential",
"ubuntu:focal::arch64": "build-essential"
},
"bzip2": {
- "debian:10": [
+ "debian:11": [
"bzip2",
"libbz2-dev"
],
@@ -54,7 +54,7 @@
]
},
"clang": {
- "debian:10": "clang",
+ "debian:11": "clang",
"ubuntu:focal": "clang",
"ubuntu:noble": "clang",
"ubuntu:focal::arch64": "clang",
@@ -66,7 +66,7 @@
"ubuntu:focal::arch64": "cmake"
},
"curl": {
- "debian:10": [
+ "debian:11": [
"curl",
"libcurl4-openssl-dev"
],
@@ -88,7 +88,7 @@
]
},
"doxygen": {
- "debian:10": "doxygen",
+ "debian:11": "doxygen",
"ubuntu:focal": "doxygen",
"ubuntu:noble": "doxygen",
"ubuntu:focal::arch64": "doxygen"
@@ -97,7 +97,7 @@
"rockylinux:8": "dnf"
},
"fuse": {
- "debian:10": [
+ "debian:11": [
"fuse",
"libfuse-dev"
],
@@ -120,7 +120,7 @@
]
},
"gcc": {
- "debian:10": {
+ "debian:11": {
"bullseye": [
"gcc",
"g++"
@@ -140,39 +140,39 @@
]
},
"git": {
- "debian:10": "git",
+ "debian:11": "git",
"ubuntu:focal": "git",
"ubuntu:noble": "git",
"ubuntu:focal::arch64": "git",
"rockylinux:8": "git"
},
"gnupg-agent": {
- "debian:10": "gnupg-agent",
+ "debian:11": "gnupg-agent",
"ubuntu:focal": "gnupg-agent",
"ubuntu:noble": "gnupg-agent",
"ubuntu:focal::arch64": "gnupg-agent"
},
"hugo": {
- "debian:10": "hugo",
+ "debian:11": "hugo",
"ubuntu:focal": "hugo",
"ubuntu:noble": "hugo",
"ubuntu:focal::arch64": "hugo"
},
"libbcprov-java": {
- "debian:10": "libbcprov-java",
+ "debian:11": "libbcprov-java",
"ubuntu:focal": "libbcprov-java",
"ubuntu:noble": "libbcprov-java",
"ubuntu:focal::arch64": "libbcprov-java"
},
"libtool": {
- "debian:10": "libtool",
+ "debian:11": "libtool",
"ubuntu:focal": "libtool",
"ubuntu:noble": "libtool",
"ubuntu:focal::arch64": "libtool",
"rockylinux:8": "libtool"
},
"openssl": {
- "debian:10": "libssl-dev",
+ "debian:11": "libssl-dev",
"ubuntu:focal": "libssl-dev",
"ubuntu:noble": "libssl-dev",
"ubuntu:focal::arch64": "libssl-dev",
@@ -185,7 +185,7 @@
]
},
"protocol-buffers": {
- "debian:10": [
+ "debian:11": [
"libprotobuf-dev",
"libprotoc-dev"
],
@@ -203,20 +203,20 @@
]
},
"sasl": {
- "debian:10": "libsasl2-dev",
+ "debian:11": "libsasl2-dev",
"ubuntu:focal": "libsasl2-dev",
"ubuntu:noble": "libsasl2-dev",
"ubuntu:focal::arch64": "libsasl2-dev",
"rockylinux:8": "cyrus-sasl-devel"
},
"snappy": {
- "debian:10": "libsnappy-dev",
+ "debian:11": "libsnappy-dev",
"ubuntu:focal": "libsnappy-dev",
"ubuntu:noble": "libsnappy-dev",
"ubuntu:focal::arch64": "libsnappy-dev"
},
"zlib": {
- "debian:10": [
+ "debian:11": [
"libzstd-dev",
"zlib1g-dev"
],
@@ -238,7 +238,7 @@
]
},
"locales": {
- "debian:10": "locales",
+ "debian:11": "locales",
"ubuntu:focal": "locales",
"ubuntu:noble": "locales",
"ubuntu:focal::arch64": "locales"
@@ -248,20 +248,23 @@
"ubuntu:noble": "libtirpc-dev"
},
"make": {
- "debian:10": "make",
+ "debian:11": "make",
"ubuntu:focal": "make",
"ubuntu:noble": "make",
"ubuntu:focal::arch64": "make",
"rockylinux:8": "make"
},
"maven": {
- "debian:10": "maven",
+ "debian:11": "maven",
"ubuntu:focal": "maven",
"ubuntu:noble": "maven",
"ubuntu:focal::arch64": "maven"
},
"java": {
- "debian:10": "openjdk-11-jdk",
+ "debian:11": [
+ "openjdk-11-jdk",
+ "openjdk-17-jdk"
+ ],
"ubuntu:focal": [
"temurin-24-jdk",
"openjdk-8-jdk",
@@ -281,21 +284,21 @@
]
},
"pinentry-curses": {
- "debian:10": "pinentry-curses",
+ "debian:11": "pinentry-curses",
"ubuntu:focal": "pinentry-curses",
"ubuntu:noble": "pinentry-curses",
"ubuntu:focal::arch64": "pinentry-curses",
"rockylinux:8": "pinentry-curses"
},
"pkg-config": {
- "debian:10": "pkg-config",
+ "debian:11": "pkg-config",
"ubuntu:focal": "pkg-config",
"ubuntu:noble": "pkg-config",
"ubuntu:focal::arch64": "pkg-config",
"rockylinux:8": "pkg-config"
},
"python": {
- "debian:10": [
+ "debian:11": [
"python3",
"python3-pip",
"python3-pkg-resources",
@@ -332,14 +335,14 @@
]
},
"rsync": {
- "debian:10": "rsync",
+ "debian:11": "rsync",
"ubuntu:focal": "rsync",
"ubuntu:noble": "rsync",
"ubuntu:focal::arch64": "rsync",
"rockylinux:8": "rsync"
},
"shellcheck": {
- "debian:10": "shellcheck",
+ "debian:11": "shellcheck",
"ubuntu:focal": "shellcheck",
"ubuntu:noble": "shellcheck",
"ubuntu:focal::arch64": "shellcheck"
@@ -348,27 +351,27 @@
"rockylinux:8": "perl-Digest-SHA"
},
"software-properties-common": {
- "debian:10": "software-properties-common",
+ "debian:11": "software-properties-common",
"ubuntu:focal": "software-properties-common",
"ubuntu:noble": "software-properties-common",
"ubuntu:focal::arch64": "software-properties-common"
},
"sudo": {
- "debian:10": "sudo",
+ "debian:11": "sudo",
"ubuntu:focal": "sudo",
"ubuntu:noble": "sudo",
"ubuntu:focal::arch64": "sudo",
"rockylinux:8": "sudo"
},
"valgrind": {
- "debian:10": "valgrind",
+ "debian:11": "valgrind",
"ubuntu:focal": "valgrind",
"ubuntu:noble": "valgrind",
"ubuntu:focal::arch64": "valgrind",
"rockylinux:8": "valgrind"
},
"yasm": {
- "debian:10": "yasm",
+ "debian:11": "yasm",
"ubuntu:focal": "yasm",
"ubuntu:noble": "yasm",
"ubuntu:focal::arch64": "yasm"
diff --git a/dev-support/docker/pkg-resolver/platforms.json b/dev-support/docker/pkg-resolver/platforms.json
index eebf7fc948f..00f6c129b20 100644
--- a/dev-support/docker/pkg-resolver/platforms.json
+++ b/dev-support/docker/pkg-resolver/platforms.json
@@ -3,5 +3,5 @@
"ubuntu:focal::arch64",
"ubuntu:noble",
"rockylinux:8",
- "debian:10"
+ "debian:11"
]
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 08/09: HADOOP-19661. (Follow-up) Install missing deps for Rocky Linux 8 docker image (#7931) Contributed by Cheng Pan.
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 0d00b7885f91087aa8e6bc6a814a495d0157cf6d
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Fri Sep 5 14:54:36 2025 +0800
HADOOP-19661. (Follow-up) Install missing deps for Rocky Linux 8 docker image (#7931) Contributed by Cheng Pan.
* HADOOP-19661. (Follow-up) Install missing deps for Rocky Linux 8 docker image
Reviewed-by: Shilun Fan <sl...@apache.org>
Signed-off-by: Shilun Fan <sl...@apache.org>
---
dev-support/docker/pkg-resolver/packages.json | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 452ed015c32..08b5cb6607e 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -23,7 +23,8 @@
"debian:11": "bats",
"ubuntu:focal": "bats",
"ubuntu:noble": "bats",
- "ubuntu:focal::arch64": "bats"
+ "ubuntu:focal::arch64": "bats",
+ "rockylinux:8": "bats"
},
"build-essential": {
"debian:11": "build-essential",
@@ -179,7 +180,7 @@
"rockylinux:8": "openssl-devel"
},
"perl": {
- "rockylinux:7": [
+ "rockylinux:8": [
"perl-CPAN",
"perl-devel"
]
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 06/09: HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8 (#7917)
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a5c8631d0cc112893aa6cadbe9cdf6f758e63514
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Fri Aug 29 14:26:09 2025 +0800
HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8 (#7917)
HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8.
Signed-off-by: Shilun Fan <sl...@apache.org>
---
dev-support/Jenkinsfile | 2 +-
dev-support/docker/{Dockerfile_centos_8 => Dockerfile_rockylinux_8} | 0
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 9ed78f1ff32..2a2b52b20f9 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -116,7 +116,7 @@ pipeline {
environment {
SOURCEDIR = "${WORKSPACE}/rockylinux-8/src"
PATCHDIR = "${WORKSPACE}/rockylinux-8/out"
- DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_centos_8"
+ DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_rockylinux_8"
IS_OPTIONAL = 1
}
diff --git a/dev-support/docker/Dockerfile_centos_8 b/dev-support/docker/Dockerfile_rockylinux_8
similarity index 100%
rename from dev-support/docker/Dockerfile_centos_8
rename to dev-support/docker/Dockerfile_rockylinux_8
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 02/09: HADOOP-19540. [JDK17] Add ubuntu:noble as a build platform with JDK-17 as default (#7608)
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 2a6df7ecc19d02c2bbb83bbbc5323ddd8d781d27
Author: Vinayakumar B <vi...@apache.org>
AuthorDate: Wed May 7 02:34:06 2025 +0800
HADOOP-19540. [JDK17] Add ubuntu:noble as a build platform with JDK-17 as default (#7608)
1. Added "ubuntu:noble" to supported docker platforms.
2. Modified `start-build-env.sh` to use expected Dockerfiles dynamically.
ex: `bash start-build-env.sh ubuntu_24` where `ubuntu_24` is the extra suffix in the name of the Dockerfile.
3. Added `jdk17++` profile activated when jdk17 is available.
Successfully built the entire Hadoop tar with native support on env created using `bash start-build-env.sh ubuntu_24`
---
BUILDING.txt | 6 +-
dev-support/docker/Dockerfile_ubuntu_24 | 106 +++++++++++++++++++++++++
dev-support/docker/pkg-resolver/packages.json | 66 ++++++++++++++-
dev-support/docker/pkg-resolver/platforms.json | 1 +
hadoop-project/pom.xml | 10 +++
start-build-env.sh | 24 +++++-
6 files changed, 208 insertions(+), 5 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index 173cc0bcad4..8795880c552 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -29,7 +29,11 @@ This requires a recent version of docker (1.4.1 and higher are known to work).
On Linux / Mac:
Install Docker and run this command:
- $ ./start-build-env.sh
+ $ ./start-build-env.sh [OS platform]
+
+ - [OS Platform] One of [centos_7, centos_8, debian_10, ubuntu_20, ubuntu_24, windows_10].
+ Default is 'ubuntu_20'.
+ Note: Currently only default ('ubuntu_20') is supported on arm machine
The prompt which is then presented is located at a mounted version of the source tree
and all required tools for testing and building have been installed and configured.
diff --git a/dev-support/docker/Dockerfile_ubuntu_24 b/dev-support/docker/Dockerfile_ubuntu_24
new file mode 100644
index 00000000000..28ab04c3e09
--- /dev/null
+++ b/dev-support/docker/Dockerfile_ubuntu_24
@@ -0,0 +1,106 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:noble
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Platform package dependency resolver
+######
+COPY pkg-resolver pkg-resolver
+RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
+ && chmod a+r pkg-resolver/*.json
+
+######
+# Install packages from apt
+######
+# hadolint ignore=DL3008,SC2046
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends wget apt-transport-https gpg gpg-agent gawk ca-certificates \
+ && apt-get -q install -y --no-install-recommends python3 \
+ && echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list \
+ && wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public > /etc/apt/trusted.gpg.d/adoptium.asc \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py ubuntu:noble) \
+ && apt-get clean \
+ && update-java-alternatives -s java-1.17.0-openjdk-amd64 \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN locale-gen en_US.UTF-8
+ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
+ENV PYTHONIOENCODING=utf-8
+
+######
+# Set env vars required to build Hadoop
+######
+ENV MAVEN_HOME /usr
+# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
+ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
+
+#######
+# Set env vars for SpotBugs 4.2.2
+#######
+ENV SPOTBUGS_HOME /opt/spotbugs
+
+#######
+# Set env vars for Google Protobuf 3.21.12
+#######
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
+
+###
+# Avoid out of memory errors in builds
+###
+ENV MAVEN_OPTS -Xms256m -Xmx3072m
+
+# Skip gpg verification when downloading Yetus via yetus-wrapper
+ENV HADOOP_SKIP_YETUS_VERIFICATION true
+
+####
+# Install packages
+####
+RUN pkg-resolver/install-spotbugs.sh ubuntu:noble
+RUN pkg-resolver/install-boost.sh ubuntu:noble
+RUN pkg-resolver/install-protobuf.sh ubuntu:noble
+RUN pkg-resolver/install-hadolint.sh ubuntu:noble
+RUN pkg-resolver/install-intel-isa-l.sh ubuntu:noble
+
+###
+# Everything past this point is either not needed for testing or breaks Yetus.
+# So tell Yetus not to read the rest of the file:
+# YETUS CUT HERE
+###
+
+# Add a welcome message and environment checks.
+COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh
+RUN chmod 755 /root/hadoop_env_checks.sh
+# hadolint ignore=SC2016
+RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 87ee435703c..4b37ea62b42 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -3,17 +3,20 @@
"debian:10": "ant",
"ubuntu:focal": "ant",
"ubuntu:focal::arch64": "ant",
+ "ubuntu:noble": "ant",
"centos:7": "ant",
"centos:8": "ant"
},
"apt-utils": {
"debian:10": "apt-utils",
"ubuntu:focal": "apt-utils",
+ "ubuntu:noble": "apt-utils",
"ubuntu:focal::arch64": "apt-utils"
},
"automake": {
"debian:10": "automake",
"ubuntu:focal": "automake",
+ "ubuntu:noble": "automake",
"ubuntu:focal::arch64": "automake",
"centos:7": "automake",
"centos:8": "automake"
@@ -24,11 +27,13 @@
"bats": {
"debian:10": "bats",
"ubuntu:focal": "bats",
+ "ubuntu:noble": "bats",
"ubuntu:focal::arch64": "bats"
},
"build-essential": {
"debian:10": "build-essential",
"ubuntu:focal": "build-essential",
+ "ubuntu:noble": "build-essential",
"ubuntu:focal::arch64": "build-essential",
"centos:7": "build-essential"
},
@@ -41,6 +46,10 @@
"bzip2",
"libbz2-dev"
],
+ "ubuntu:noble": [
+ "bzip2",
+ "libbz2-dev"
+ ],
"ubuntu:focal::arch64": [
"bzip2",
"libbz2-dev"
@@ -57,12 +66,14 @@
"clang": {
"debian:10": "clang",
"ubuntu:focal": "clang",
+ "ubuntu:noble": "clang",
"ubuntu:focal::arch64": "clang",
"centos:7": "clang",
"centos:8": "clang"
},
"cmake": {
"ubuntu:focal": "cmake",
+ "ubuntu:noble": "cmake",
"ubuntu:focal::arch64": "cmake"
},
"curl": {
@@ -74,6 +85,10 @@
"curl",
"libcurl4-openssl-dev"
],
+ "ubuntu:noble": [
+ "curl",
+ "libcurl4-openssl-dev"
+ ],
"ubuntu:focal::arch64": [
"curl",
"libcurl4-openssl-dev"
@@ -90,6 +105,7 @@
"doxygen": {
"debian:10": "doxygen",
"ubuntu:focal": "doxygen",
+ "ubuntu:noble": "doxygen",
"ubuntu:focal::arch64": "doxygen",
"centos:7": "doxygen"
},
@@ -105,6 +121,10 @@
"fuse",
"libfuse-dev"
],
+ "ubuntu:noble": [
+ "fuse",
+ "libfuse-dev"
+ ],
"ubuntu:focal::arch64": [
"fuse",
"libfuse-dev"
@@ -131,6 +151,10 @@
"gcc",
"g++"
],
+ "ubuntu:noble": [
+ "gcc",
+ "g++"
+ ],
"ubuntu:focal::arch64": [
"gcc",
"g++"
@@ -146,27 +170,32 @@
"git": {
"debian:10": "git",
"ubuntu:focal": "git",
+ "ubuntu:noble": "git",
"ubuntu:focal::arch64": "git",
"centos:8": "git"
},
"gnupg-agent": {
"debian:10": "gnupg-agent",
"ubuntu:focal": "gnupg-agent",
+ "ubuntu:noble": "gnupg-agent",
"ubuntu:focal::arch64": "gnupg-agent"
},
"hugo": {
"debian:10": "hugo",
"ubuntu:focal": "hugo",
+ "ubuntu:noble": "hugo",
"ubuntu:focal::arch64": "hugo"
},
"libbcprov-java": {
"debian:10": "libbcprov-java",
"ubuntu:focal": "libbcprov-java",
+ "ubuntu:noble": "libbcprov-java",
"ubuntu:focal::arch64": "libbcprov-java"
},
"libtool": {
"debian:10": "libtool",
"ubuntu:focal": "libtool",
+ "ubuntu:noble": "libtool",
"ubuntu:focal::arch64": "libtool",
"centos:7": "libtool",
"centos:8": "libtool"
@@ -174,6 +203,7 @@
"openssl": {
"debian:10": "libssl-dev",
"ubuntu:focal": "libssl-dev",
+ "ubuntu:noble": "libssl-dev",
"ubuntu:focal::arch64": "libssl-dev",
"centos:7": "openssl-devel",
"centos:8": "openssl-devel"
@@ -193,6 +223,10 @@
"libprotobuf-dev",
"libprotoc-dev"
],
+ "ubuntu:noble": [
+ "libprotobuf-dev",
+ "libprotoc-dev"
+ ],
"ubuntu:focal::arch64": [
"libprotobuf-dev",
"libprotoc-dev"
@@ -201,6 +235,7 @@
"sasl": {
"debian:10": "libsasl2-dev",
"ubuntu:focal": "libsasl2-dev",
+ "ubuntu:noble": "libsasl2-dev",
"ubuntu:focal::arch64": "libsasl2-dev",
"centos:7": "cyrus-sasl-devel",
"centos:8": "cyrus-sasl-devel"
@@ -208,6 +243,7 @@
"snappy": {
"debian:10": "libsnappy-dev",
"ubuntu:focal": "libsnappy-dev",
+ "ubuntu:noble": "libsnappy-dev",
"ubuntu:focal::arch64": "libsnappy-dev",
"centos:7": "snappy-devel"
},
@@ -220,6 +256,10 @@
"libzstd-dev",
"zlib1g-dev"
],
+ "ubuntu:noble": [
+ "libzstd-dev",
+ "zlib1g-dev"
+ ],
"ubuntu:focal::arch64": [
"libzstd-dev",
"zlib1g-dev"
@@ -236,11 +276,13 @@
"locales": {
"debian:10": "locales",
"ubuntu:focal": "locales",
+ "ubuntu:noble": "locales",
"ubuntu:focal::arch64": "locales"
},
"libtirpc-devel": {
"centos:7": "libtirpc-devel",
- "centos:8": "libtirpc-devel"
+ "centos:8": "libtirpc-devel",
+ "ubuntu:noble": "libtirpc-dev"
},
"libpmem": {
"centos:7": "libpmem-devel"
@@ -248,6 +290,7 @@
"make": {
"debian:10": "make",
"ubuntu:focal": "make",
+ "ubuntu:noble": "make",
"ubuntu:focal::arch64": "make",
"centos:7": "make",
"centos:8": "make"
@@ -255,6 +298,7 @@
"maven": {
"debian:10": "maven",
"ubuntu:focal": "maven",
+ "ubuntu:noble": "maven",
"ubuntu:focal::arch64": "maven"
},
"java": {
@@ -265,6 +309,11 @@
"openjdk-11-jdk",
"openjdk-17-jdk"
],
+ "ubuntu:noble": [
+ "temurin-24-jdk",
+ "openjdk-11-jdk",
+ "openjdk-17-jdk"
+ ],
"ubuntu:focal::arch64": [
"temurin-24-jdk",
"openjdk-8-jdk",
@@ -275,6 +324,7 @@
"pinentry-curses": {
"debian:10": "pinentry-curses",
"ubuntu:focal": "pinentry-curses",
+ "ubuntu:noble": "pinentry-curses",
"ubuntu:focal::arch64": "pinentry-curses",
"centos:7": "pinentry-curses",
"centos:8": "pinentry-curses"
@@ -282,6 +332,7 @@
"pkg-config": {
"debian:10": "pkg-config",
"ubuntu:focal": "pkg-config",
+ "ubuntu:noble": "pkg-config",
"ubuntu:focal::arch64": "pkg-config",
"centos:8": "pkg-config"
},
@@ -300,6 +351,13 @@
"python3-setuptools",
"python3-wheel"
],
+ "ubuntu:noble": [
+ "python3",
+ "python3-pip",
+ "python3-pkg-resources",
+ "python3-setuptools",
+ "python3-wheel"
+ ],
"ubuntu:focal::arch64": [
"python2.7",
"python3",
@@ -324,6 +382,7 @@
"rsync": {
"debian:10": "rsync",
"ubuntu:focal": "rsync",
+ "ubuntu:noble": "rsync",
"ubuntu:focal::arch64": "rsync",
"centos:7": "rsync",
"centos:8": "rsync"
@@ -331,6 +390,7 @@
"shellcheck": {
"debian:10": "shellcheck",
"ubuntu:focal": "shellcheck",
+ "ubuntu:noble": "shellcheck",
"ubuntu:focal::arch64": "shellcheck"
},
"shasum": {
@@ -340,11 +400,13 @@
"software-properties-common": {
"debian:10": "software-properties-common",
"ubuntu:focal": "software-properties-common",
+ "ubuntu:noble": "software-properties-common",
"ubuntu:focal::arch64": "software-properties-common"
},
"sudo": {
"debian:10": "sudo",
"ubuntu:focal": "sudo",
+ "ubuntu:noble": "sudo",
"ubuntu:focal::arch64": "sudo",
"centos:7": "sudo",
"centos:8": "sudo"
@@ -352,6 +414,7 @@
"valgrind": {
"debian:10": "valgrind",
"ubuntu:focal": "valgrind",
+ "ubuntu:noble": "valgrind",
"ubuntu:focal::arch64": "valgrind",
"centos:7": "valgrind",
"centos:8": "valgrind"
@@ -359,6 +422,7 @@
"yasm": {
"debian:10": "yasm",
"ubuntu:focal": "yasm",
+ "ubuntu:noble": "yasm",
"ubuntu:focal::arch64": "yasm"
}
}
diff --git a/dev-support/docker/pkg-resolver/platforms.json b/dev-support/docker/pkg-resolver/platforms.json
index 93e2a93df42..2ff65b52823 100644
--- a/dev-support/docker/pkg-resolver/platforms.json
+++ b/dev-support/docker/pkg-resolver/platforms.json
@@ -1,6 +1,7 @@
[
"ubuntu:focal",
"ubuntu:focal::arch64",
+ "ubuntu:noble",
"centos:7",
"centos:8",
"debian:10"
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 251d86965e3..0a4cda6ffc3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -2786,6 +2786,16 @@
</dependencies>
</dependencyManagement>
</profile>
+ <profile>
+ <id>jdk17+</id>
+ <activation>
+ <jdk>[17,)</jdk>
+ </activation>
+ <properties>
+ <javac.version>17</javac.version>
+ <maven.compiler.release>${javac.version}</maven.compiler.release>
+ </properties>
+ </profile>
</profiles>
<repositories>
diff --git a/start-build-env.sh b/start-build-env.sh
index 375a027a06b..de82d7e64ab 100755
--- a/start-build-env.sh
+++ b/start-build-env.sh
@@ -19,12 +19,29 @@ set -e # exit on error
cd "$(dirname "$0")" # connect to root
+OS_PLATFORM="${1:-}"
+[ "$#" -gt 0 ] && shift
+
+DEFAULT_OS_PLATFORM="ubuntu_20"
+
+OS_PLATFORM_SUFFIX=""
+
+if [[ -n ${OS_PLATFORM} ]] && [[ "${OS_PLATFORM}" != "${DEFAULT_OS_PLATFORM}" ]]; then
+ # ubuntu_20 (default) platform does not have suffix in Dockerfile.
+ OS_PLATFORM_SUFFIX="_${OS_PLATFORM}"
+fi
+
DOCKER_DIR=dev-support/docker
DOCKER_FILE="${DOCKER_DIR}/Dockerfile"
CPU_ARCH=$(echo "$MACHTYPE" | cut -d- -f1)
-if [[ "$CPU_ARCH" = "aarch64" || "$CPU_ARCH" = "arm64" ]]; then
- DOCKER_FILE="${DOCKER_DIR}/Dockerfile_aarch64"
+if [[ "$CPU_ARCH" == "aarch64" || "$CPU_ARCH" == "arm64" ]]; then
+ DOCKER_FILE="${DOCKER_DIR}/Dockerfile${OS_PLATFORM_SUFFIX}_aarch64"
+fi
+
+if [ ! -e "${DOCKER_FILE}" ] ; then
+ echo "'${OS_PLATFORM}' environment not available yet for '${CPU_ARCH}'"
+ exit 1
fi
docker build -t hadoop-build -f $DOCKER_FILE $DOCKER_DIR
@@ -93,4 +110,5 @@ docker run --rm=true $DOCKER_INTERACTIVE_RUN \
-v "${HOME}/.m2:${DOCKER_HOME_DIR}/.m2${V_OPTS:-}" \
-v "${HOME}/.gnupg:${DOCKER_HOME_DIR}/.gnupg${V_OPTS:-}" \
-u "${USER_ID}" \
- "hadoop-build-${USER_ID}" "$@"
+ --name "hadoop-build${OS_PLATFORM_SUFFIX}" \
+ "hadoop-build${OS_PLATFORM_SUFFIX}-${USER_ID}" "$@"
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 09/09: HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image (#7932) Contributed by Cheng Pan.
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 721bdabe3cba01fb8c3a1f97ca039f6daf4a088d
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Sun Sep 14 15:13:23 2025 +0800
HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image (#7932) Contributed by Cheng Pan.
* HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image.
Signed-off-by: Shilun Fan <sl...@apache.org>
---
dev-support/docker/pkg-resolver/packages.json | 3 +++
1 file changed, 3 insertions(+)
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 08b5cb6607e..b46d10e8200 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -203,6 +203,9 @@
"libprotoc-dev"
]
},
+ "procps": {
+ "debian:11": "procps"
+ },
"sasl": {
"debian:11": "libsasl2-dev",
"ubuntu:focal": "libsasl2-dev",
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 04/09: HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. (#7822)
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit ffdb112acfe4131e8dad60107285e99cf2521e91
Author: slfan1989 <55...@users.noreply.github.com>
AuthorDate: Tue Aug 26 10:28:19 2025 +0800
HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. (#7822)
* HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile.
Co-authored-by: Cheng Pan <ch...@apache.org>
Co-authored-by: Istvan Toth <st...@apache.org>
Reviewed-by: Xiaoqiao He <he...@apache.org>
Reviewed-by: Steve Loughran <st...@apache.org>
Reviewed-by: Chris Nauroth <cn...@apache.org>
Reviewed-by: Cheng Pan <ch...@apache.org>
Reviewed-by: Istvan Toth <st...@apache.org>
Signed-off-by: Shilun Fan <sl...@apache.org>
---
dev-support/Jenkinsfile | 52 ------------
dev-support/docker/Dockerfile_centos_7 | 108 -------------------------
dev-support/docker/pkg-resolver/packages.json | 58 +------------
dev-support/docker/pkg-resolver/platforms.json | 1 -
4 files changed, 3 insertions(+), 216 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 51225268b65..06095cc8a4f 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -83,12 +83,6 @@ pipeline {
// optional stages after the first one.
stage ('setup sources') {
steps {
- dir("${WORKSPACE}/centos-7") {
- sh '''#!/usr/bin/env bash
-
- cp -Rp ${WORKSPACE}/src ${WORKSPACE}/centos-7
- '''
- }
dir("${WORKSPACE}/centos-8") {
sh '''#!/usr/bin/env bash
@@ -113,52 +107,6 @@ pipeline {
}
}
- // This is an optional stage which runs only when there's a change in
- // C++/C++ build/platform.
- // This stage serves as a means of cross platform validation, which is
- // really needed to ensure that any C++ related/platform change doesn't
- // break the Hadoop build on Centos 7.
- stage ('precommit-run Centos 7') {
- environment {
- SOURCEDIR = "${WORKSPACE}/centos-7/src"
- PATCHDIR = "${WORKSPACE}/centos-7/out"
- DOCKERFILE = "${SOURCEDIR}/dev-support/docker/Dockerfile_centos_7"
- IS_OPTIONAL = 1
- }
-
- steps {
- withCredentials(getGithubCreds()) {
- sh '''#!/usr/bin/env bash
-
- chmod u+x "${SOURCEDIR}/dev-support/jenkins.sh"
- "${SOURCEDIR}/dev-support/jenkins.sh" run_ci
- '''
- }
- }
-
- post {
- // Since this is an optional platform, we want to copy the artifacts
- // and archive it only if the build fails, to help with debugging.
- failure {
- sh '''#!/usr/bin/env bash
-
- cp -Rp "${WORKSPACE}/centos-7/out" "${WORKSPACE}"
- '''
- archiveArtifacts "out/**"
- }
-
- cleanup() {
- script {
- sh '''#!/usr/bin/env bash
-
- chmod u+x "${SOURCEDIR}/dev-support/jenkins.sh"
- "${SOURCEDIR}/dev-support/jenkins.sh" cleanup_ci_proc
- '''
- }
- }
- }
- }
-
// This is an optional stage which runs only when there's a change in
// C++/C++ build/platform.
// This stage serves as a means of cross platform validation, which is
diff --git a/dev-support/docker/Dockerfile_centos_7 b/dev-support/docker/Dockerfile_centos_7
deleted file mode 100644
index b97e59969a7..00000000000
--- a/dev-support/docker/Dockerfile_centos_7
+++ /dev/null
@@ -1,108 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Dockerfile for installing the necessary dependencies for building Hadoop.
-# See BUILDING.txt.
-
-FROM centos:7
-
-WORKDIR /root
-
-SHELL ["/bin/bash", "-o", "pipefail", "-c"]
-
-######
-# Platform package dependency resolver
-######
-COPY pkg-resolver pkg-resolver
-RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
- && chmod a+r pkg-resolver/*.json
-
-######
-# Centos 7 has reached its EOL and the packages
-# are no longer available on mirror.centos.org site.
-# Please see https://www.centos.org/centos-linux-eol/
-######
-RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7
-
-######
-# Install packages from yum
-######
-# hadolint ignore=DL3008,SC2046
-RUN yum update -y \
- && yum groupinstall -y "Development Tools" \
- && yum install -y \
- centos-release-scl \
- python3
-
-# Apply the script again because centos-release-scl creates new YUM repo files
-RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7
-
-# hadolint ignore=DL3008,SC2046
-RUN yum install -y $(pkg-resolver/resolve.py centos:7)
-
-# Set GCC 9 as the default C/C++ compiler
-RUN echo "source /opt/rh/devtoolset-9/enable" >> /etc/bashrc
-SHELL ["/bin/bash", "--login", "-c"]
-
-######
-# Set the environment variables needed for CMake
-# to find and use GCC 9 for compilation
-######
-ENV GCC_HOME "/opt/rh/devtoolset-9"
-ENV CC "${GCC_HOME}/root/usr/bin/gcc"
-ENV CXX "${GCC_HOME}/root/usr/bin/g++"
-ENV SHLVL 1
-ENV LD_LIBRARY_PATH "${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:${GCC_HOME}/root/usr/lib64/dyninst:${GCC_HOME}/root/usr/lib/dyninst:${GCC_HOME}/root/usr/lib64:${GCC_HOME}/root/usr/lib:/usr/lib:/usr/lib64"
-ENV PCP_DIR "${GCC_HOME}/root"
-ENV MANPATH "${GCC_HOME}/root/usr/share/man:"
-ENV PATH "${GCC_HOME}/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
-ENV PKG_CONFIG_PATH "${GCC_HOME}/root/usr/lib64/pkgconfig"
-ENV INFOPATH "${GCC_HOME}/root/usr/share/info"
-
-# TODO: Set locale
-
-######
-# Set env vars required to build Hadoop
-######
-ENV MAVEN_HOME /opt/maven
-ENV PATH "${PATH}:${MAVEN_HOME}/bin"
-# JAVA_HOME must be set in Maven >= 3.5.0 (MNG-6003)
-ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
-
-#######
-# Set env vars for SpotBugs
-#######
-ENV SPOTBUGS_HOME /opt/spotbugs
-
-#######
-# Set env vars for Google Protobuf 3.21.12
-#######
-ENV PROTOBUF_HOME /opt/protobuf
-ENV PATH "${PATH}:/opt/protobuf/bin"
-
-######
-# Install packages
-######
-RUN pkg-resolver/install-maven.sh centos:7
-RUN pkg-resolver/install-cmake.sh centos:7
-RUN pkg-resolver/install-zstandard.sh centos:7
-RUN pkg-resolver/install-yasm.sh centos:7
-RUN pkg-resolver/install-protobuf.sh centos:7
-RUN pkg-resolver/install-boost.sh centos:7
-RUN pkg-resolver/install-spotbugs.sh centos:7
-RUN pkg-resolver/install-nodejs.sh centos:7
-RUN pkg-resolver/install-git.sh centos:7
-RUN pkg-resolver/install-common-pkgs.sh
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 4b37ea62b42..efa6b1726ce 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -4,7 +4,6 @@
"ubuntu:focal": "ant",
"ubuntu:focal::arch64": "ant",
"ubuntu:noble": "ant",
- "centos:7": "ant",
"centos:8": "ant"
},
"apt-utils": {
@@ -18,12 +17,8 @@
"ubuntu:focal": "automake",
"ubuntu:noble": "automake",
"ubuntu:focal::arch64": "automake",
- "centos:7": "automake",
"centos:8": "automake"
},
- "autoconf": {
- "centos:7": "autoconf"
- },
"bats": {
"debian:10": "bats",
"ubuntu:focal": "bats",
@@ -34,8 +29,7 @@
"debian:10": "build-essential",
"ubuntu:focal": "build-essential",
"ubuntu:noble": "build-essential",
- "ubuntu:focal::arch64": "build-essential",
- "centos:7": "build-essential"
+ "ubuntu:focal::arch64": "build-essential"
},
"bzip2": {
"debian:10": [
@@ -54,10 +48,6 @@
"bzip2",
"libbz2-dev"
],
- "centos:7": [
- "bzip2",
- "bzip2-devel"
- ],
"centos:8": [
"bzip2",
"bzip2-devel"
@@ -68,7 +58,6 @@
"ubuntu:focal": "clang",
"ubuntu:noble": "clang",
"ubuntu:focal::arch64": "clang",
- "centos:7": "clang",
"centos:8": "clang"
},
"cmake": {
@@ -93,10 +82,6 @@
"curl",
"libcurl4-openssl-dev"
],
- "centos:7": [
- "curl",
- "libcurl-devel"
- ],
"centos:8": [
"curl",
"libcurl-devel"
@@ -106,8 +91,7 @@
"debian:10": "doxygen",
"ubuntu:focal": "doxygen",
"ubuntu:noble": "doxygen",
- "ubuntu:focal::arch64": "doxygen",
- "centos:7": "doxygen"
+ "ubuntu:focal::arch64": "doxygen"
},
"dnf": {
"centos:8": "dnf"
@@ -129,11 +113,6 @@
"fuse",
"libfuse-dev"
],
- "centos:7": [
- "fuse",
- "fuse-libs",
- "fuse-devel"
- ],
"centos:8": [
"fuse",
"fuse-libs",
@@ -158,15 +137,8 @@
"ubuntu:focal::arch64": [
"gcc",
"g++"
- ],
- "centos:7": [
- "centos-release-scl",
- "devtoolset-9"
]
},
- "gettext": {
- "centos:7": "gettext-devel"
- },
"git": {
"debian:10": "git",
"ubuntu:focal": "git",
@@ -197,7 +169,6 @@
"ubuntu:focal": "libtool",
"ubuntu:noble": "libtool",
"ubuntu:focal::arch64": "libtool",
- "centos:7": "libtool",
"centos:8": "libtool"
},
"openssl": {
@@ -205,7 +176,6 @@
"ubuntu:focal": "libssl-dev",
"ubuntu:noble": "libssl-dev",
"ubuntu:focal::arch64": "libssl-dev",
- "centos:7": "openssl-devel",
"centos:8": "openssl-devel"
},
"perl": {
@@ -237,15 +207,13 @@
"ubuntu:focal": "libsasl2-dev",
"ubuntu:noble": "libsasl2-dev",
"ubuntu:focal::arch64": "libsasl2-dev",
- "centos:7": "cyrus-sasl-devel",
"centos:8": "cyrus-sasl-devel"
},
"snappy": {
"debian:10": "libsnappy-dev",
"ubuntu:focal": "libsnappy-dev",
"ubuntu:noble": "libsnappy-dev",
- "ubuntu:focal::arch64": "libsnappy-dev",
- "centos:7": "snappy-devel"
+ "ubuntu:focal::arch64": "libsnappy-dev"
},
"zlib": {
"debian:10": [
@@ -264,10 +232,6 @@
"libzstd-dev",
"zlib1g-dev"
],
- "centos:7": [
- "zlib-devel",
- "lz4-devel"
- ],
"centos:8": [
"zlib-devel",
"lz4-devel"
@@ -280,19 +244,14 @@
"ubuntu:focal::arch64": "locales"
},
"libtirpc-devel": {
- "centos:7": "libtirpc-devel",
"centos:8": "libtirpc-devel",
"ubuntu:noble": "libtirpc-dev"
},
- "libpmem": {
- "centos:7": "libpmem-devel"
- },
"make": {
"debian:10": "make",
"ubuntu:focal": "make",
"ubuntu:noble": "make",
"ubuntu:focal::arch64": "make",
- "centos:7": "make",
"centos:8": "make"
},
"maven": {
@@ -326,7 +285,6 @@
"ubuntu:focal": "pinentry-curses",
"ubuntu:noble": "pinentry-curses",
"ubuntu:focal::arch64": "pinentry-curses",
- "centos:7": "pinentry-curses",
"centos:8": "pinentry-curses"
},
"pkg-config": {
@@ -366,12 +324,6 @@
"python3-setuptools",
"python3-wheel"
],
- "centos:7": [
- "python3",
- "python3-pip",
- "python3-setuptools",
- "python3-wheel"
- ],
"centos:8": [
"python3",
"python3-pip",
@@ -384,7 +336,6 @@
"ubuntu:focal": "rsync",
"ubuntu:noble": "rsync",
"ubuntu:focal::arch64": "rsync",
- "centos:7": "rsync",
"centos:8": "rsync"
},
"shellcheck": {
@@ -394,7 +345,6 @@
"ubuntu:focal::arch64": "shellcheck"
},
"shasum": {
- "centos:7": "perl-Digest-SHA",
"centos:8": "perl-Digest-SHA"
},
"software-properties-common": {
@@ -408,7 +358,6 @@
"ubuntu:focal": "sudo",
"ubuntu:noble": "sudo",
"ubuntu:focal::arch64": "sudo",
- "centos:7": "sudo",
"centos:8": "sudo"
},
"valgrind": {
@@ -416,7 +365,6 @@
"ubuntu:focal": "valgrind",
"ubuntu:noble": "valgrind",
"ubuntu:focal::arch64": "valgrind",
- "centos:7": "valgrind",
"centos:8": "valgrind"
},
"yasm": {
diff --git a/dev-support/docker/pkg-resolver/platforms.json b/dev-support/docker/pkg-resolver/platforms.json
index 2ff65b52823..47842cac9b4 100644
--- a/dev-support/docker/pkg-resolver/platforms.json
+++ b/dev-support/docker/pkg-resolver/platforms.json
@@ -2,7 +2,6 @@
"ubuntu:focal",
"ubuntu:focal::arch64",
"ubuntu:noble",
- "centos:7",
"centos:8",
"debian:10"
]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
(hadoop) 01/09: HADOOP-19495. Add JDK 24 to Ubuntu 20.04 docker development images.
Posted by sl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b793e111d821ded97d5a203d34e3d5102c1e1f0a
Author: Istvan Toth <st...@apache.org>
AuthorDate: Wed Mar 26 01:27:25 2025 +0800
HADOOP-19495. Add JDK 24 to Ubuntu 20.04 docker development images.
---
dev-support/docker/Dockerfile | 5 +++++
dev-support/docker/Dockerfile_aarch64 | 5 +++++
dev-support/docker/pkg-resolver/packages.json | 2 ++
3 files changed, 12 insertions(+)
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index 3b71e622a57..6beef860a9b 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -44,9 +44,14 @@ RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
######
# hadolint ignore=DL3008,SC2046
RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends wget apt-transport-https gpg gpg-agent gawk ca-certificates \
&& apt-get -q install -y --no-install-recommends python3 \
+ && echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list \
+ && wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public > /etc/apt/trusted.gpg.d/adoptium.asc \
+ && apt-get -q update \
&& apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py ubuntu:focal) \
&& apt-get clean \
+ && update-java-alternatives -s java-1.8.0-openjdk-amd64 \
&& rm -rf /var/lib/apt/lists/*
RUN locale-gen en_US.UTF-8
diff --git a/dev-support/docker/Dockerfile_aarch64 b/dev-support/docker/Dockerfile_aarch64
index 26ad10edd6f..b9249f5aec9 100644
--- a/dev-support/docker/Dockerfile_aarch64
+++ b/dev-support/docker/Dockerfile_aarch64
@@ -44,10 +44,15 @@ RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \
######
# hadolint ignore=DL3008,SC2046
RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends wget apt-transport-https gpg gpg-agent gawk ca-certificates \
&& apt-get -q install -y --no-install-recommends python3 \
&& apt-get -q install -y --no-install-recommends phantomjs \
+ && echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list \
+ && wget -q -O - https://packages.adoptium.net/artifactory/api/gpg/key/public > /etc/apt/trusted.gpg.d/adoptium.asc \
+ && apt-get -q update \
&& apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py ubuntu:focal::arch64) \
&& apt-get clean \
+ && update-java-alternatives -s java-1.8.0-openjdk-arm64 \
&& rm -rf /var/lib/apt/lists/*
RUN locale-gen en_US.UTF-8
diff --git a/dev-support/docker/pkg-resolver/packages.json b/dev-support/docker/pkg-resolver/packages.json
index 00d6ba36de2..87ee435703c 100644
--- a/dev-support/docker/pkg-resolver/packages.json
+++ b/dev-support/docker/pkg-resolver/packages.json
@@ -260,11 +260,13 @@
"java": {
"debian:10": "openjdk-11-jdk",
"ubuntu:focal": [
+ "temurin-24-jdk",
"openjdk-8-jdk",
"openjdk-11-jdk",
"openjdk-17-jdk"
],
"ubuntu:focal::arch64": [
+ "temurin-24-jdk",
"openjdk-8-jdk",
"openjdk-11-jdk",
"openjdk-17-jdk"
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org