You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/09 12:09:00 UTC
[GitHub] [hadoop] vinayakumarb opened a new pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
vinayakumarb opened a new pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801
## What does this change do?
Added a Dockerfile, which will be used by scripts during build and test in aarch64 platform.
### Following are the differences between x86 and aarch64 Dockerfile
-----
* CMake 3.1.0 compilation from source as no binary package readily available for aarch64
* Skips installation of ghc package which does not have any package for aarch64 from repo.
* Skips installation of hadolint as no package is available for aarch64
* x86_64 specific packages were replaced with aarch64 specific packages.
* phantomjs is being installed from prebuilt package and made available in the system path to avoid failure during maven build.
### Additional changes
* Updated the *dev-support/bin/createrelease* and *start-build-env.sh* script to load platform specific docker files.
* Updated *dev-support/bin/createrelease* to make use of gpg-agent for signing artifacts.
### Testing ?
* Yes, Tested well!!
Verified creation of release package using *dev-support/bin/createrelease* itself in aarch64 platform.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r365104130
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
Review comment:
> Could be fixed in both places in a separate jira?
Yes, it could be fixed in a separate jira. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r365083893
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
Review comment:
Can we use cmake 3.5.1 that Xenial ships?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572534432
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/console in case of problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r364855538
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
+######
+# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial)
+######
+# hadolint ignore=DL3003
+RUN mkdir -p /opt/protobuf-src \
+ && curl -L -s -S \
+ https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \
+ -o /opt/protobuf.tar.gz \
+ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \
+ && cd /opt/protobuf-src \
+ && ./configure --prefix=/opt/protobuf \
+ && make install \
+ && cd /root \
+ && rm -rf /opt/protobuf-src
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
+
+######
+# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends maven \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV MAVEN_HOME /usr
+
+######
+# Install findbugs 3.0.1 (3.0.1 ships with Xenial)
+# Ant is needed for findbugs
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends findbugs ant \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV FINDBUGS_HOME /usr
+
+####
+# Install shellcheck (0.4.6, the latest as of 2017-09-26)
+####
+# hadolint ignore=DL3008
+RUN add-apt-repository -y ppa:hvr/ghc \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends shellcheck \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial)
+####
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends bats \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install pylint at fixed version (2.0.0 removed python2 support)
+# https://github.com/PyCQA/pylint/issues/2294
+####
+RUN pip2 install pylint==1.9.2
+
+####
+# Install dateutil.parser
+####
+RUN pip2 install python-dateutil==2.7.3
+
+###
+# Install node.js 8.17.0 for web UI framework (4.2.6 ships with Xenial)
+###
+RUN curl -L -s -S https://deb.nodesource.com/setup_8.x | bash - \
+ && apt-get install -y --no-install-recommends nodejs=8.17.0-1nodesource1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ && npm install -g bower@1.8.8
+
+###
+## Install Yarn 1.12.1 for web UI framework
+####
+RUN curl -s -S https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+ && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list \
+ && apt-get -q update \
+ && apt-get install -y --no-install-recommends yarn=1.21.1-1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+###
+# Install hadolint
+####
+#RUN curl -L -s -S \
+# https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \
Review comment:
Does the X86 one works for ARM too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r364862130
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
+######
+# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial)
+######
+# hadolint ignore=DL3003
+RUN mkdir -p /opt/protobuf-src \
+ && curl -L -s -S \
+ https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \
+ -o /opt/protobuf.tar.gz \
+ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \
+ && cd /opt/protobuf-src \
+ && ./configure --prefix=/opt/protobuf \
+ && make install \
+ && cd /root \
+ && rm -rf /opt/protobuf-src
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
+
+######
+# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends maven \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV MAVEN_HOME /usr
+
+######
+# Install findbugs 3.0.1 (3.0.1 ships with Xenial)
+# Ant is needed for findbugs
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends findbugs ant \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV FINDBUGS_HOME /usr
+
+####
+# Install shellcheck (0.4.6, the latest as of 2017-09-26)
+####
+# hadolint ignore=DL3008
+RUN add-apt-repository -y ppa:hvr/ghc \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends shellcheck \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial)
+####
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends bats \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install pylint at fixed version (2.0.0 removed python2 support)
+# https://github.com/PyCQA/pylint/issues/2294
+####
+RUN pip2 install pylint==1.9.2
+
+####
+# Install dateutil.parser
+####
+RUN pip2 install python-dateutil==2.7.3
+
+###
+# Install node.js 8.17.0 for web UI framework (4.2.6 ships with Xenial)
+###
+RUN curl -L -s -S https://deb.nodesource.com/setup_8.x | bash - \
+ && apt-get install -y --no-install-recommends nodejs=8.17.0-1nodesource1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ && npm install -g bower@1.8.8
+
+###
+## Install Yarn 1.12.1 for web UI framework
+####
+RUN curl -s -S https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+ && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list \
+ && apt-get -q update \
+ && apt-get install -y --no-install-recommends yarn=1.21.1-1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+###
+# Install hadolint
+####
+#RUN curl -L -s -S \
+# https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \
+# -o /bin/hadolint \
+# && chmod a+rx /bin/hadolint \
+# && shasum -a 512 /bin/hadolint | \
+# awk '$1!="734e37c1f6619cbbd86b9b249e69c9af8ee1ea87a2b1ff71dccda412e9dac35e63425225a95d71572091a3f0a11e9a04c2fc25d9e91b840530c26af32b9891ca" {exit(1)}'
+
+###
+# Install phantomjs built for aarch64
+####
+RUN mkdir -p /opt/phantomjs \
+ && curl -L -s -S \
+ https://github.com/liusheng/phantomjs/releases/download/2.1.1/phantomjs-2.1.1-linux-aarch64.tar.bz2 \
+ -o /opt/phantomjs/phantomjs-2.1.1-linux-aarch64.tar.bz2 \
+ && tar xvjf /opt/phantomjs/phantomjs-2.1.1-linux-aarch64.tar.bz2 --strip-components 1 -C /opt/phantomjs \
+ && cp /opt/phantomjs/bin/phantomjs /usr/bin/ \
+ && rm -rf /opt/phantomjs
+
+###
+# Avoid out of memory errors in builds
+###
+ENV MAVEN_OPTS -Xms256m -Xmx1536m
+
+###
+# Everything past this point is either not needed for testing or breaks Yetus.
+# So tell Yetus not to read the rest of the file:
+# YETUS CUT HERE
+###
+
+# Hugo static website generator (for new hadoop site and Ozone docs)
Review comment:
Need not to mention ozone here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572925149
@ayushtkn Here is the final output summary of the below command
```
$ dev-support/bin/create-release --native --rc-label=aarch64-rc1 --docker
..
..
============================================================================
============================================================================
Hadoop Release Creator
============================================================================
============================================================================
Version to create : 3.3.0-SNAPSHOT
Release Candidate Label: aarch64-rc1
Source Version : 3.3.0-SNAPSHOT
```
Attached the complete output.
[create-release-output.log](https://github.com/apache/hadoop/files/4044593/create-release-output.log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572905420
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/2/console in case of problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572663720
Yetus, has complains, Pls give a check.
Would it be possible for you to share one Build report from an Arm machine?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-573074538
Thanx @vinayakumarb seems the build got aborted, Pls check once
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572594259
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 15s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 35s | trunk passed |
| +1 :green_heart: | compile | 16m 43s | trunk passed |
| +1 :green_heart: | mvnsite | 16m 14s | trunk passed |
| +1 :green_heart: | shadedclient | 12m 39s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 7m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 19m 9s | the patch passed |
| +1 :green_heart: | compile | 16m 21s | the patch passed |
| +1 :green_heart: | javac | 16m 21s | the patch passed |
| -1 :x: | hadolint | 0m 2s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 16m 15s | the patch passed |
| -1 :x: | shellcheck | 0m 1s | The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| -1 :x: | shadedclient | 3m 13s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 7m 3s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 17m 42s | root in the patch failed. |
| +1 :green_heart: | asflicense | 0m 58s | The patch does not generate ASF License warnings. |
| | | 158m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1801 |
| Optional Tests | dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient xml |
| uname | Linux dd0e18049ed0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a40dc9e |
| Default Java | 1.8.0_232 |
| hadolint | https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/diff-patch-hadolint.txt |
| shellcheck | https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/diff-patch-shellcheck.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/artifact/out/patch-unit-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/testReport/ |
| Max. process+thread count | 1373 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs . U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/1/console |
| versions | git=2.7.4 maven=3.3.9 shellcheck=0.3.7 hadolint=1.11.1-0-g0e692dd |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-573112682
Yetus detected that change done in root of the project and chose to run unit tests of all modules. This needs approx 15 hours instead of current time-out of 5 hours.
So build got aborted.
But other than this every thing else should be fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r365085230
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
+######
+# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial)
+######
+# hadolint ignore=DL3003
+RUN mkdir -p /opt/protobuf-src \
+ && curl -L -s -S \
+ https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \
+ -o /opt/protobuf.tar.gz \
+ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \
+ && cd /opt/protobuf-src \
+ && ./configure --prefix=/opt/protobuf \
+ && make install \
+ && cd /root \
+ && rm -rf /opt/protobuf-src
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
+
+######
+# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends maven \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV MAVEN_HOME /usr
+
+######
+# Install findbugs 3.0.1 (3.0.1 ships with Xenial)
+# Ant is needed for findbugs
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends findbugs ant \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV FINDBUGS_HOME /usr
+
+####
+# Install shellcheck (0.4.6, the latest as of 2017-09-26)
+####
+# hadolint ignore=DL3008
+RUN add-apt-repository -y ppa:hvr/ghc \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends shellcheck \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial)
+####
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends bats \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install pylint at fixed version (2.0.0 removed python2 support)
+# https://github.com/PyCQA/pylint/issues/2294
+####
+RUN pip2 install pylint==1.9.2
+
+####
+# Install dateutil.parser
+####
+RUN pip2 install python-dateutil==2.7.3
+
+###
+# Install node.js 8.17.0 for web UI framework (4.2.6 ships with Xenial)
+###
+RUN curl -L -s -S https://deb.nodesource.com/setup_8.x | bash - \
+ && apt-get install -y --no-install-recommends nodejs=8.17.0-1nodesource1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ && npm install -g bower@1.8.8
+
+###
+## Install Yarn 1.12.1 for web UI framework
+####
+RUN curl -s -S https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+ && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list \
+ && apt-get -q update \
+ && apt-get install -y --no-install-recommends yarn=1.21.1-1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+###
+# Install hadolint
+####
+#RUN curl -L -s -S \
+# https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \
+# -o /bin/hadolint \
+# && chmod a+rx /bin/hadolint \
+# && shasum -a 512 /bin/hadolint | \
+# awk '$1!="734e37c1f6619cbbd86b9b249e69c9af8ee1ea87a2b1ff71dccda412e9dac35e63425225a95d71572091a3f0a11e9a04c2fc25d9e91b840530c26af32b9891ca" {exit(1)}'
+
+###
+# Install phantomjs built for aarch64
+####
+RUN mkdir -p /opt/phantomjs \
+ && curl -L -s -S \
+ https://github.com/liusheng/phantomjs/releases/download/2.1.1/phantomjs-2.1.1-linux-aarch64.tar.bz2 \
+ -o /opt/phantomjs/phantomjs-2.1.1-linux-aarch64.tar.bz2 \
+ && tar xvjf /opt/phantomjs/phantomjs-2.1.1-linux-aarch64.tar.bz2 --strip-components 1 -C /opt/phantomjs \
+ && cp /opt/phantomjs/bin/phantomjs /usr/bin/ \
+ && rm -rf /opt/phantomjs
+
+###
+# Avoid out of memory errors in builds
+###
+ENV MAVEN_OPTS -Xms256m -Xmx1536m
+
+###
+# Everything past this point is either not needed for testing or breaks Yetus.
+# So tell Yetus not to read the rest of the file:
+# YETUS CUT HERE
+###
+
+# Hugo static website generator (for new hadoop site and Ozone docs)
Review comment:
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-573510558
Merged.
Thanks @ayushtkn and @aajisaka for reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#issuecomment-572926568
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/hadoop-multibranch/job/PR-1801/3/console in case of problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb merged pull request #1801: HADOOP-16797. Add
Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb merged pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r365084704
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
+######
+# Install Google Protobuf 3.7.1 (2.6.0 ships with Xenial)
+######
+# hadolint ignore=DL3003
+RUN mkdir -p /opt/protobuf-src \
+ && curl -L -s -S \
+ https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \
+ -o /opt/protobuf.tar.gz \
+ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \
+ && cd /opt/protobuf-src \
+ && ./configure --prefix=/opt/protobuf \
+ && make install \
+ && cd /root \
+ && rm -rf /opt/protobuf-src
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
+
+######
+# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends maven \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV MAVEN_HOME /usr
+
+######
+# Install findbugs 3.0.1 (3.0.1 ships with Xenial)
+# Ant is needed for findbugs
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends findbugs ant \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV FINDBUGS_HOME /usr
+
+####
+# Install shellcheck (0.4.6, the latest as of 2017-09-26)
+####
+# hadolint ignore=DL3008
+RUN add-apt-repository -y ppa:hvr/ghc \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends shellcheck \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial)
+####
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends bats \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+####
+# Install pylint at fixed version (2.0.0 removed python2 support)
+# https://github.com/PyCQA/pylint/issues/2294
+####
+RUN pip2 install pylint==1.9.2
+
+####
+# Install dateutil.parser
+####
+RUN pip2 install python-dateutil==2.7.3
+
+###
+# Install node.js 8.17.0 for web UI framework (4.2.6 ships with Xenial)
+###
+RUN curl -L -s -S https://deb.nodesource.com/setup_8.x | bash - \
+ && apt-get install -y --no-install-recommends nodejs=8.17.0-1nodesource1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/* \
+ && npm install -g bower@1.8.8
+
+###
+## Install Yarn 1.12.1 for web UI framework
+####
+RUN curl -s -S https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
+ && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list \
+ && apt-get -q update \
+ && apt-get install -y --no-install-recommends yarn=1.21.1-1 \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+###
+# Install hadolint
+####
+#RUN curl -L -s -S \
+# https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \
Review comment:
These lines are copied from x86 docker file and since there was no alternative available for aarch64 just commented the line.
This will be required when precommit builds are configured to run on aarch64.
So these lines will be ingored while creating the image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on a change in pull request #1801:
HADOOP-16797. Add Dockerfile for ARM builds
Posted by GitBox <gi...@apache.org>.
vinayakumarb commented on a change in pull request #1801: HADOOP-16797. Add Dockerfile for ARM builds
URL: https://github.com/apache/hadoop/pull/1801#discussion_r365085627
##########
File path: dev-support/docker/Dockerfile_aarch64
##########
@@ -0,0 +1,244 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dockerfile for installing the necessary dependencies for building Hadoop.
+# See BUILDING.txt.
+
+FROM ubuntu:xenial
+
+WORKDIR /root
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_TERSE true
+
+######
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ clang \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ valgrind \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+#######
+# OpenJDK 8
+#######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-8-jdk libbcprov-java \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+# There is no cmake binary available for aarch64. Build from source.
+######
+RUN mkdir -p /opt/cmake/src \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-1-src.tar.bz2 \
+ -o /opt/cmake/cmake-src.tar.bz2 \
+ && tar xvjf /opt/cmake/cmake-src.tar.bz2 -C /opt/cmake/src \
+ && cd /opt/cmake/src \
+ && tar xvjf cmake-3.1.0.tar.bz2 \
+ && cd cmake-3.1.0 && patch -p0 -i ../cmake-3.1.0-1.patch && mkdir .build && cd .build \
+ && ../bootstrap --parallel=2 \
+ && make -j2 && ./bin/cpack \
+ && tar xzf cmake-3.1.0-Linux-aarch64.tar.gz --strip-components 1 -C /opt/cmake \
+ && cd /opt/cmake && rm -rf /opt/cmake/src
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
+
Review comment:
I am not sure about it. I will give a check.
x86 docker also uses 3.1.0 CMake itself.
Could be fixed in both places in a separate jira?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org