You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by cl...@apache.org on 2019/07/12 20:07:16 UTC
[hadoop] branch branch-3.1 updated (773591e -> 2a1451a)
This is an automated email from the ASF dual-hosted git repository.
cliang pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 773591e YARN-9626. UI2 - Fair scheduler queue apps page issues. Contributed by Zoltan Siegl
new 4cf6390 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun.
new 21a886a HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
new b999e96 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
new 07a6510 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
new 18e545f HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun.
new 7d0c3ba HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
new c40ddf9 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen.
new abb8614 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
new 8e55170 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen.
new 405f6b6 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
new 9840dae HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
new cfb4399 HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov.
new ac2e793 HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen.
new 849a6c6 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
new d96ddb3 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
new ee9977c HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
new 1ad1c80 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 1be8e18 HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 504cb70 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
new 8ec17f6 HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
new e639297 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov.
new 0755862 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
new d4adf92 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
new 91803c8 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
new a054c65 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko.
new e860882 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko.
new 66db260 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng.
new 8100dcb HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
new e0436b3 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang.
new 53a88bf HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
new f5c923b HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
new c4d944a HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
new 6cb1b5a HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
new 6e54d97 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
new 265bd68 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
new 8e77349 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
new 67c4768 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
new c23755a HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng.
new 6c1bf61 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
new 7a650b0 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
new 8f36bdc HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
new b8f9739 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
new b028d62 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun.
new 321e26c HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
new ee0416b HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
new 98684fb HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
new 35200b3 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
new 769b5a8 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
new cc22373 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
new 3ba7826 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar.
new 2a1451a HDFS-13791. Amend fix LOG string format.
The 51 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/ha/FailoverController.java | 2 +-
.../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++
.../org/apache/hadoop/ha/HAServiceProtocol.java | 18 +
.../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 +
.../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 +
.../HAServiceProtocolClientSideTranslatorPB.java | 16 +
.../HAServiceProtocolServerSideTranslatorPB.java | 20 +
.../hadoop/io/retry/RetryInvocationHandler.java | 5 +
.../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++
.../main/java/org/apache/hadoop/ipc/Client.java | 40 +-
.../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +-
.../hadoop/ipc/ObserverRetryOnActiveException.java | 37 ++
.../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +-
.../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++-
.../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 147 ++++++-
.../org/apache/hadoop/ipc/WritableRpcEngine.java | 47 ++-
.../org/apache/hadoop/log/LogThrottlingHelper.java | 34 ++
.../java/org/apache/hadoop/util/ProtoUtil.java | 13 +
.../src/main/proto/HAServiceProtocol.proto | 20 +
.../hadoop-common/src/main/proto/RpcHeader.proto | 2 +
.../hadoop-common/src/site/markdown/Metrics.md | 5 +
.../java/org/apache/hadoop/ha/DummyHAService.java | 18 +-
.../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 +
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +-
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 ++++
.../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++
.../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +-
.../hadoop/hdfs/protocol/ClientProtocol.java | 67 +++
.../ClientNamenodeProtocolTranslatorPB.java | 37 ++
.../ha/AbstractNNFailoverProxyProvider.java | 19 +-
.../server/namenode/ha/ClientHAProxyFactory.java | 12 +
.../namenode/ha/IPFailoverProxyProvider.java | 2 +-
.../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++
.../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++
.../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++
.../src/main/proto/ClientNamenodeProtocol.proto | 18 +
.../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 +++++
.../resolver/FederationNamenodeServiceState.java | 3 +
.../federation/router/RouterClientProtocol.java | 13 +
.../server/federation/router/RouterRpcServer.java | 12 +
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 +
...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 +
.../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 +
.../hdfs/qjournal/client/AsyncLoggerSet.java | 14 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 137 +++++-
.../hdfs/qjournal/client/QuorumOutputStream.java | 13 +-
.../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +-
.../QJournalProtocolServerSideTranslatorPB.java | 14 +
.../protocolPB/QJournalProtocolTranslatorPB.java | 20 +
.../hadoop/hdfs/qjournal/server/Journal.java | 75 +++-
.../hdfs/qjournal/server/JournalMetrics.java | 20 +-
.../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 +
.../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++
.../hdfs/server/common/HdfsServerConstants.java | 4 +-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../server/namenode/EditLogFileInputStream.java | 44 ++
.../hdfs/server/namenode/FSEditLogLoader.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSImage.java | 19 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 31 +-
.../hdfs/server/namenode/GlobalStateIdContext.java | 158 +++++++
.../hadoop/hdfs/server/namenode/NameNode.java | 41 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 23 +-
.../namenode/RedundantEditLogInputStream.java | 15 +-
.../hdfs/server/namenode/ha/EditLogTailer.java | 8 +-
.../hdfs/server/namenode/ha/StandbyState.java | 19 +-
.../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 +
.../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 +
.../src/main/proto/QJournalProtocol.proto | 18 +
.../src/main/resources/hdfs-default.xml | 16 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../site/markdown/HDFSHighAvailabilityWithQJM.md | 42 +-
.../src/site/markdown/ObserverNameNode.md | 231 ++++++++++
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +-
.../hdfs/TestStateAlignmentContextWithHA.java | 347 +++++++++++++++
.../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +-
.../qjournal/client/TestQuorumJournalManager.java | 134 ++++++
.../client/TestQuorumJournalManagerUnit.java | 103 ++++-
.../hdfs/qjournal/server/JournalTestUtil.java | 48 +++
.../hadoop/hdfs/qjournal/server/TestJournal.java | 46 ++
.../qjournal/server/TestJournaledEditsCache.java | 257 +++++++++++
.../server/namenode/NNThroughputBenchmark.java | 6 +-
.../hdfs/server/namenode/NameNodeAdapter.java | 13 +
.../namenode/TestEditLogFileInputStream.java | 18 +
.../hdfs/server/namenode/TestFSEditLogLoader.java | 48 +++
.../hadoop/hdfs/server/namenode/TestFsck.java | 2 +-
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 143 ++++++-
.../namenode/ha/TestConsistentReadsObserver.java | 385 +++++++++++++++++
.../namenode/ha/TestDelegationTokensWithHA.java | 46 ++
.../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++
.../hdfs/server/namenode/ha/TestObserverNode.java | 364 ++++++++++++++++
.../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++
.../namenode/ha/TestStandbyInProgressTail.java | 164 +++++--
.../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +-
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++
.../yarn/server/resourcemanager/AdminService.java | 7 +
99 files changed, 5940 insertions(+), 163 deletions(-)
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org