You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/02/02 01:20:21 UTC
[hadoop] branch branch-2-jhung-test updated (e566e52 -> b3e8e72)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch branch-2-jhung-test
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard e566e52 java 8
discard 1720bc2 install openjdk7 too
discard 5fd64d7 HADOOP-15610. Fixed pylint version for Hadoop docker image. Contributed by Jack Bearden
discard d3fd3e5 HADOOP-14816. Update Dockerfile to use Xenial. Contributed by Allen Wittenauer
discard b78304a test
discard d10a214 test
discard 407fb5b add jvm flag
discard 34c17ed change params
discard 8af12f4 Ignore setrep tests
discard 95a3ae2 testing2
discard e9d3e1a test
discard 3c984df testing
add 300d65c YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma.
add a5f678f HADOOP-15617. Node.js and npm package loading in the Dockerfile failing on branch-2.
add d3b06d1 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian.
add c77f7e2 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang.
add c02c191 YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)
add 7bca49c HADOOP-16055. Upgrade AWS SDK to 1.11.271 in branch-2.
add 338dbbe YARN-9222. Print launchTime in ApplicationSummary
add 6d3e7a8 HADOOP-16049. DistCp result has data and checksum mismatch when blocks per chunk > 0.
add 9688ed3 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki.
add c6c5e94 YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph
new 9b7932a HADOOP-16053
new b3e8e72 Default java 7 in Dockerfile. Also install openjdk8
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e566e52)
\
N -- N -- N refs/heads/branch-2-jhung-test (b3e8e72)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
NOTICE.txt | 10 +-
dev-support/docker/Dockerfile | 208 ++++++++++++---------
dev-support/docker/hadoop_env_checks.sh | 15 +-
.../java/org/apache/hadoop/ipc/FairCallQueue.java | 32 +++-
.../hadoop-common/src/site/markdown/Metrics.md | 10 +
.../org/apache/hadoop/ipc/TestFairCallQueue.java | 36 ++++
.../hdfs/server/namenode/MetaRecoveryContext.java | 4 +-
.../offlineImageViewer/OfflineImageViewerPB.java | 3 +-
.../hadoop-hdfs/src/site/markdown/HdfsDesign.md | 5 +
.../org/apache/hadoop/hdfs/DataNodeCluster.java | 3 +-
.../apache/hadoop/hdfs/TestSetrepDecreasing.java | 2 -
.../apache/hadoop/hdfs/TestSetrepIncreasing.java | 4 -
.../hadoop/hdfs/server/namenode/TestEditLog.java | 2 -
.../namenode/TestEditLogFileInputStream.java | 6 -
.../namenode/TestEditLogFileOutputStream.java | 2 -
.../hdfs/server/namenode/TestEditLogRace.java | 2 -
.../hdfs/server/namenode/TestNameNodeRecovery.java | 2 -
.../server/namenode/TestSecurityTokenEditLog.java | 2 -
.../namenode/ha/TestDelegationTokensWithHA.java | 2 -
.../hdfs/server/namenode/ha/TestHAAppend.java | 4 -
.../server/namenode/ha/TestRetryCacheWithHA.java | 2 -
.../offlineImageViewer/TestOfflineImageViewer.java | 2 -
.../org/apache/hadoop/mapred/NotRunningJob.java | 2 +-
.../hadoop/mapred/TestClientServiceDelegate.java | 4 +-
.../org/apache/hadoop/mapred/TestYARNRunner.java | 3 +-
hadoop-project/pom.xml | 10 +-
.../tools/mapred/RetriableFileCopyCommand.java | 24 ++-
.../hadoop/tools/util/ThrottledInputStream.java | 48 ++---
.../org/apache/hadoop/tools/TestDistCpSync.java | 49 ++++-
.../hadoop/tools/TestDistCpSyncReverseBase.java | 102 ++++++----
.../apache/hadoop/tools/mapred/TestCopyMapper.java | 24 ++-
.../hadoop/yarn/api/records/ApplicationReport.java | 90 ++++++++-
.../src/main/proto/yarn_protos.proto | 2 +
.../hadoop/yarn/client/ProtocolHATestBase.java | 2 +-
.../hadoop/yarn/client/api/impl/TestAHSClient.java | 10 +-
.../yarn/client/api/impl/TestYarnClient.java | 8 +-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 42 ++---
.../records/impl/pb/ApplicationReportPBImpl.java | 23 +++
.../hadoop/yarn/api/TestApplicatonReport.java | 2 +-
.../ApplicationHistoryManagerImpl.java | 6 +-
.../ApplicationHistoryManagerOnTimelineStore.java | 20 +-
...stApplicationHistoryManagerOnTimelineStore.java | 1 +
.../webapp/TestAHSWebServices.java | 1 +
.../hadoop/yarn/server/utils/BuilderUtils.java | 3 +-
.../apache/hadoop/yarn/server/webapp/AppBlock.java | 7 +-
.../hadoop/yarn/server/webapp/AppsBlock.java | 5 +-
.../hadoop/yarn/server/webapp/WebPageUtils.java | 2 +-
.../hadoop/yarn/server/webapp/dao/AppInfo.java | 10 +-
.../yarn/server/resourcemanager/RMAppManager.java | 1 +
.../server/resourcemanager/ResourceManager.java | 1 -
.../resourcemanager/amlauncher/AMLauncher.java | 2 +-
.../recovery/records/ApplicationStateData.java | 24 ++-
.../impl/pb/ApplicationStateDataPBImpl.java | 13 ++
.../yarn/server/resourcemanager/rmapp/RMApp.java | 10 +-
.../server/resourcemanager/rmapp/RMAppEvent.java | 6 +
.../resourcemanager/rmapp/RMAppEventType.java | 1 +
.../server/resourcemanager/rmapp/RMAppImpl.java | 41 +++-
.../rmapp/attempt/RMAppAttemptEvent.java | 7 +
.../rmapp/attempt/RMAppAttemptImpl.java | 4 +-
.../webapp/FairSchedulerAppsBlock.java | 2 +
.../server/resourcemanager/webapp/NodesPage.java | 5 +-
.../server/resourcemanager/webapp/RMAppsBlock.java | 2 +
.../server/resourcemanager/webapp/dao/AppInfo.java | 6 +
.../yarn_server_resourcemanager_recovery.proto | 1 +
.../server/resourcemanager/TestAppManager.java | 2 +
.../applicationsmanager/MockAsm.java | 25 ++-
.../recovery/RMStateStoreTestBase.java | 4 +-
.../recovery/TestFSRMStateStore.java | 2 +-
.../recovery/TestZKRMStateStore.java | 2 +-
.../server/resourcemanager/rmapp/MockRMApp.java | 6 +
.../rmapp/TestRMAppTransitions.java | 2 +-
.../resourcemanager/webapp/TestRMWebApp.java | 2 +-
.../webapp/TestRMWebServicesApps.java | 14 +-
.../yarn/server/router/webapp/JavaProcess.java | 2 +-
.../hadoop-yarn-ui/src/main/webapp/bower.json | 2 +-
75 files changed, 740 insertions(+), 307 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/02: Default java 7 in Dockerfile. Also install openjdk8
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-2-jhung-test
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b3e8e72cd57f4eef7cf9b600a0468a289998e778
Author: Jonathan Hung <jh...@linkedin.com>
AuthorDate: Fri Feb 1 17:19:54 2019 -0800
Default java 7 in Dockerfile. Also install openjdk8
---
dev-support/docker/Dockerfile | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index 6594c8b..9d3b6bf 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -90,6 +90,12 @@ RUN apt-get -q update \
&& apt-get -q install -y --no-install-recommends openjdk-7-jdk \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+RUN update-java-alternatives --set java-1.7.0-openjdk-amd64
+
+######
+# OpenJDK 8
+######
+RUN apt-get -q install -y openjdk-8-jdk
######
# Install cmake 3.1.0 (3.5.1 ships with Xenial)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/02: HADOOP-16053
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch branch-2-jhung-test
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 9b7932a2ec2a3aa477d252f721e731e7913cefb3
Author: Jonathan Hung <jh...@linkedin.com>
AuthorDate: Fri Feb 1 17:12:47 2019 -0800
HADOOP-16053
---
dev-support/docker/Dockerfile | 264 +++++++++++++++++++++++++++---------------
1 file changed, 169 insertions(+), 95 deletions(-)
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index f6dc6e4..6594c8b 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -18,150 +18,224 @@
# Dockerfile for installing the necessary dependencies for building Hadoop.
# See BUILDING.txt.
-
-FROM ubuntu:trusty
+FROM ubuntu:xenial
WORKDIR /root
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+#####
+# Disable suggests/recommends
+#####
+RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras
+RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras
+
ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_TERSE true
######
-# Install common dependencies from packages
-######
-RUN apt-get -q update && apt-get -q install --no-install-recommends -y \
- ant \
- build-essential \
- bzip2 \
- cmake \
- curl \
- doxygen \
- fuse \
- g++ \
- gcc \
- git \
- gnupg-agent \
- make \
- maven \
- libbz2-dev \
- libcurl4-openssl-dev \
- libfuse-dev \
- libjansson-dev \
- libprotobuf-dev \
- libprotoc-dev \
- libsnappy-dev \
- libssl-dev \
- libtool \
- openjdk-7-jdk \
- pinentry-curses \
- pkg-config \
- protobuf-compiler \
- protobuf-c-compiler \
- python \
- python2.7 \
- python2.7-dev \
- python-pip \
- rsync \
- snappy \
- zlib1g-dev
-
-######
-# Install Apache Maven
+# Install common dependencies from packages. Versions here are either
+# sufficient or irrelevant.
+#
+# WARNING: DO NOT PUT JAVA APPS HERE! Otherwise they will install default
+# Ubuntu Java. See Java section below!
######
-RUN mkdir -p /opt/maven && \
- curl -L -s -S \
- http://www-us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz \
- -o /opt/maven.tar.gz && \
- tar xzf /opt/maven.tar.gz --strip-components 1 -C /opt/maven
-ENV MAVEN_HOME /opt/maven
-
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends \
+ apt-utils \
+ build-essential \
+ bzip2 \
+ curl \
+ doxygen \
+ fuse \
+ g++ \
+ gcc \
+ git \
+ gnupg-agent \
+ libbz2-dev \
+ libcurl4-openssl-dev \
+ libfuse-dev \
+ libprotobuf-dev \
+ libprotoc-dev \
+ libsasl2-dev \
+ libsnappy-dev \
+ libssl-dev \
+ libtool \
+ libzstd1-dev \
+ locales \
+ make \
+ pinentry-curses \
+ pkg-config \
+ python \
+ python2.7 \
+ python-pip \
+ python-pkg-resources \
+ python-setuptools \
+ python-wheel \
+ rsync \
+ software-properties-common \
+ snappy \
+ sudo \
+ zlib1g-dev \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
#######
-# Java OpenJDK
+# OpenJDK 7
#######
+# hadolint ignore=DL3008
+RUN add-apt-repository ppa:openjdk-r/ppa
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends openjdk-7-jdk \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
-RUN echo "dot_style = mega" > "/root/.wgetrc"
-RUN echo "quiet = on" >> "/root/.wgetrc"
+######
+# Install cmake 3.1.0 (3.5.1 ships with Xenial)
+######
+RUN mkdir -p /opt/cmake \
+ && curl -L -s -S \
+ https://cmake.org/files/v3.1/cmake-3.1.0-Linux-x86_64.tar.gz \
+ -o /opt/cmake.tar.gz \
+ && tar xzf /opt/cmake.tar.gz --strip-components 1 -C /opt/cmake
+ENV CMAKE_HOME /opt/cmake
+ENV PATH "${PATH}:/opt/cmake/bin"
-RUN apt-get -q install --no-install-recommends -y software-properties-common
-RUN add-apt-repository -y ppa:webupd8team/java
-RUN apt-get -q update
+######
+# Install Google Protobuf 2.5.0 (2.6.0 ships with Xenial)
+######
+# hadolint ignore=DL3003
+RUN mkdir -p /opt/protobuf-src \
+ && curl -L -s -S \
+ https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz \
+ -o /opt/protobuf.tar.gz \
+ && tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src \
+ && cd /opt/protobuf-src \
+ && ./configure --prefix=/opt/protobuf \
+ && make install \
+ && cd /root \
+ && rm -rf /opt/protobuf-src
+ENV PROTOBUF_HOME /opt/protobuf
+ENV PATH "${PATH}:/opt/protobuf/bin"
-# Install OpenJDK 7
-RUN apt-get -q install --no-install-recommends -y openjdk-7-jdk
+######
+# Install Apache Maven 3.3.9 (3.3.9 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends maven \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV MAVEN_HOME /usr
######
-# Install spotbugs (successor of findbugs)
+# Install Apache Ant 1.9.6 (1.9.6 ships with Xenial)
######
-RUN mkdir -p /opt/spotbugs && \
- curl -L -s -S \
- http://repo.maven.apache.org/maven2/com/github/spotbugs/spotbugs/3.1.2/spotbugs-3.1.2.tgz \
- -o /opt/spotbugs.tar.gz && \
- tar xzf /opt/spotbugs.tar.gz --strip-components 1 -C /opt/spotbugs
-# Hadoop uses FINDBUGS_HOME to run spotbugs
-ENV FINDBUGS_HOME /opt/spotbugs
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends ant \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+######
+# Install findbugs 3.0.1 (3.0.1 ships with Xenial)
+######
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends findbugs \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+ENV FINDBUGS_HOME /usr
####
-# Install shellcheck
+# Install shellcheck (0.4.6, the latest as of 2017-09-26)
####
-RUN apt-get -q install -y cabal-install
-RUN mkdir /root/.cabal
-RUN echo "remote-repo: hackage.fpcomplete.com:http://hackage.fpcomplete.com/" >> /root/.cabal/config
-#RUN echo "remote-repo: hackage.haskell.org:http://hackage.haskell.org/" > /root/.cabal/config
-RUN echo "remote-repo-cache: /root/.cabal/packages" >> /root/.cabal/config
-RUN cabal update
-RUN cabal install shellcheck --global
+# hadolint ignore=DL3008
+RUN add-apt-repository -y ppa:jonathonf/ghc-8.0.2 \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends shellcheck \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
####
-# Install bats
+# Install bats (0.4.0, the latest as of 2017-09-26, ships with Xenial)
####
-RUN add-apt-repository -y ppa:duggan/bats
-RUN apt-get -q update
-RUN apt-get -q install --no-install-recommends -y bats
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends bats \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
####
-# Install pylint
+# Install pylint at fixed version (2.0.0 removed python2 support)
+# https://github.com/PyCQA/pylint/issues/2294
####
-RUN pip install pylint==1.9.2
+RUN pip2 install pylint==1.9.2
####
# Install dateutil.parser
####
-RUN pip install python-dateutil
-
-###
-# Avoid out of memory errors in builds
-###
-ENV MAVEN_OPTS -Xms256m -Xmx1536m
+RUN pip2 install python-dateutil==2.7.3
###
-# Install node js tools for web UI frameowkr
+# Install node.js for web UI framework (4.2.6 ships with Xenial)
###
+# hadolint ignore=DL3008, DL3016
RUN apt-get -q update \
&& apt-get install -y --no-install-recommends nodejs npm \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& ln -s /usr/bin/nodejs /usr/bin/node \
- && npm config set strict-ssl false \
&& npm install npm@latest -g \
&& npm install -g jshint
###
+# Install hadolint
+####
+RUN curl -L -s -S \
+ https://github.com/hadolint/hadolint/releases/download/v1.11.1/hadolint-Linux-x86_64 \
+ -o /bin/hadolint \
+ && chmod a+rx /bin/hadolint \
+ && shasum -a 512 /bin/hadolint | \
+ awk '$1!="734e37c1f6619cbbd86b9b249e69c9af8ee1ea87a2b1ff71dccda412e9dac35e63425225a95d71572091a3f0a11e9a04c2fc25d9e91b840530c26af32b9891ca" {exit(1)}'
+
+###
+# Avoid out of memory errors in builds
+###
+ENV MAVEN_OPTS -Xms256m -Xmx1536m
+
+
+###
# Everything past this point is either not needed for testing or breaks Yetus.
# So tell Yetus not to read the rest of the file:
# YETUS CUT HERE
###
####
-# Install Forrest (for Apache Hadoop website)
+# Install svn & Forrest (for Apache Hadoop website)
###
-RUN mkdir -p /usr/local/apache-forrest ; \
- curl -s -S -O http://archive.apache.org/dist/forrest/0.8/apache-forrest-0.8.tar.gz ; \
- tar xzf *forrest* --strip-components 1 -C /usr/local/apache-forrest ; \
- echo 'forrest.home=/usr/local/apache-forrest' > build.properties
+# hadolint ignore=DL3008
+RUN apt-get -q update \
+ && apt-get -q install -y --no-install-recommends subversion \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir -p /opt/apache-forrest \
+ && curl -L -s -S \
+ https://archive.apache.org/dist/forrest/0.8/apache-forrest-0.8.tar.gz \
+ -o /opt/forrest.tar.gz \
+ && tar xzf /opt/forrest.tar.gz --strip-components 1 -C /opt/apache-forrest
+RUN echo 'forrest.home=/opt/apache-forrest' > build.properties
+ENV FORREST_HOME=/opt/apache-forrest
+
+# Hugo static website generator (for new hadoop site and Ozone docs)
+RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.30.2/hugo_0.30.2_Linux-64bit.deb \
+ && dpkg --install hugo.deb \
+ && rm hugo.deb
# Add a welcome message and environment checks.
-ADD hadoop_env_checks.sh /root/hadoop_env_checks.sh
+COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh
RUN chmod 755 /root/hadoop_env_checks.sh
-RUN echo '~/hadoop_env_checks.sh' >> /root/.bashrc
-
+# hadolint ignore=SC2016
+RUN echo '${HOME}/hadoop_env_checks.sh' >> /root/.bashrc
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org