You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/08/21 22:41:20 UTC
[hadoop] branch YARN-8200 updated (4c6b60a -> a8901a1)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard 4c6b60a YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda)
discard eaaad74 YARN-9409. Port resource type changes from YARN-7237 to branch-2
discard 8a0c434 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types
discard 10148a7 YARN-9271. Backport YARN-6927 for resource type support in MapReduce
discard ac4dee6 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
discard 44749cf YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák.
discard 44f39ae YARN-7143. FileNotFound handling in ResourceUtils is inconsistent
discard 4739fbe YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda)
discard f989415 YARN-9291. Backport YARN-7637 to branch-2
discard 0031fd5 YARN-9397. Fix empty NMResourceInfo object test failures in branch-2
discard b678fd7 YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan.
discard 1db4657 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák.
discard b6dd404 YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda)
discard e764708a YARN-9289. Backport YARN-7330 for GPU in UI to branch-2
discard bd455e8 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung
discard a70acb2 YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class
discard 2d3a421 YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation
discard 0f4b1f8 YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2
discard 27465f8 YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2
discard befd3ef YARN-9175. Null resources check in ResourceInfo for branch-3.0
discard c11feaf YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan.
discard d27dfd4 YARN-7270 addendum: Reapplied changes after YARN-3926 backports
discard fd45a43 YARN-9188. Port YARN-7136 to branch-2
discard 22bf8b7 YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)
discard ef1ebcd YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R.
discard 5f90ce6 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda)
discard cc88e4e YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda)
discard c662fba YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan.
discard 420fff3 YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda)
discard ad73704 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin.
discard a2f4b35 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
discard 774e735 YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda)
discard 1bf4885 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan.
discard 9c295d1 YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton.
discard 7964b05 YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G.
discard eeee6c7 YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu)
discard e2347bb YARN-6788. [YARN-3926] Improve performance of resource profile branch (Contributed by Sunil Govindan via Daniel Templeton)
discard 0949e62 YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
discard 1ce4f43 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
discard 81a6280 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
discard c790b18 YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2
discard 3971afc YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
discard b49b7d6 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev.
discard 5cad0b7 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh)
discard 29cbdb7 YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev.
discard ccc1c65 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev.
discard dcaedb2 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev.
discard 6f73b2b YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh)
discard 3887d36 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev.
discard 7005a25 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda)
discard cd0e044 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda)
add 1f0bf69 MAPREDUCE-6521. MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test.
add 68c24b3 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung)
add 2cbd7eb MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build
add dc4d7f6 YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
add 3bfae20 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena.
add c0fd09c HADOOP-9844. NPE when trying to create an error message response of SASL RPC
add 653951e Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena."
add edb4635 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.
add 27ead06 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai.
add 61581a8 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao.
add 46c00ac HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.
add 3941b3c HDFS-14464. Remove unnecessary log message from DFSInputStream. Contributed by Chao Sun.
add 77d1aa9 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan.
add e8ccc29 HADOOP-15237. In KMS docs there should be one space between KMS_LOG and NOTE.
add a503662 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira.
add 9aae72a YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit
add 75fc058 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
add 4e5ba31 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
add 19af228 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
add ee1547a HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao.
add 567e117 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
add 4a9fc45 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung)
add 63531be HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
add 97600cd HDFS-14631. The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun.
add ce12c8f HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun.
add 15062b6 HDFS-14696. Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class) (#1251) Contributed by Siyao Meng.
add 61b5108 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun.
add dd70e6d HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
add d0c85f8 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
add d5da529 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
add c450e69 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun.
add 3ed8d45 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
add befe410 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen.
add 88d65af HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
add 85f81fe HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou.
add 272a198 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
add 04eefe2 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
add 9803d8d HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov.
add ea6d367 HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen.
add 33d8aae HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. Also fix a issue in IPCLoggerChannel
add f531301 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
add 9f90bca HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
add fcafa72 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
add c09665d HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
add 7ccb2a4 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. Also add a missing change from HDFS-13610
add bf9d8ba HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
add 7eacf5f HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. Also fix a number of java7 incompatibilities from previous SbN read commits.
add a7d551e HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
add 7146e67 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
add b4fc93d HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
add ae559db HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko.
add 63cf980 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko.
add 37dfd29 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng.
add 40d5cc1 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
add 697207c HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang.
add 17fac44 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
add c6267bb HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
add 52d066a HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
add 3518767 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
add 3c73819 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
add 0705e83 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
add 68da1e4 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
add a86d955 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
add 49bdc53 HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng.
add 7b1a3b5 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
add 705e1c8 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
add 41e85cc HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
add 3ec2ff7 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
add 8cf1c0b HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun.
add 3804b8e HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
add b247a8b [SBN read] Addendum: Misc fix to be Java 7 compatible
add a0327f5 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
add 3b2b97a6 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
add c055e78 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
add 5190fd6 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
add c377540 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
add 2fe1505 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
add 9599c91 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar.
add 12dc7ff HDFS-12836. startTxId could be greater than endTxId when tailing in-progress edit log. Contributed by Chao Sun.
add 0977101 [SBN read] Addendum: Misc fix of issues, javadoc, whitespace unused backported fields etc
add 585b6de Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
add 0d153dc HDFS-13898 Addendum: add a missing test helper method
add 885b7f4 Fix JDK7 compatibility in branch-2.
add 719214f HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen.
add afa9a40 YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan.
add 99cd181 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
add 670de35 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
add 61bd868 HDFS-14707. Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2.
add b020c5c HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao.
add 8755de1 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow.
add ab7509e HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee.
add bdab03c HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai.
add 69b80f1 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow."
new c15354b YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda)
new fceaa96 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda)
new f3d2560 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev.
new 11c5b26 YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh)
new ab11d68 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev.
new 76d9e7c YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev.
new 77c780b YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev.
new 7e4bfa4 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh)
new 0dab70c YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev.
new d9c5672 YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
new bbd3b19 YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2
new 2384394 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
new b5dd7a7 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
new b98bc4b YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
new 5bf11f5 YARN-6788. [YARN-3926] Improve performance of resource profile branch (Contributed by Sunil Govindan via Daniel Templeton)
new 1c06445 YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu)
new b6328bc YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G.
new 3b35cda YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton.
new 5489073 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan.
new 907144b YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda)
new d86b628 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
new 3c1e31dc0 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin.
new b3d6947 YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda)
new bf3fe09 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan.
new 1d5c5f9 YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda)
new a1e7be1 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda)
new 04a0e99 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R.
new 33785f8 YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)
new 575b86a YARN-9188. Port YARN-7136 to branch-2
new 52377b0 YARN-7270 addendum: Reapplied changes after YARN-3926 backports
new 3f7d9c8 YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan.
new 51160ce YARN-9175. Null resources check in ResourceInfo for branch-3.0
new b8c0a65 YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2
new c0753ff YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2
new e53ec71 YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation
new 862a933 YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class
new a8bf897 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung
new 9d875f8 YARN-9289. Backport YARN-7330 for GPU in UI to branch-2
new 61b2a2f YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda)
new ce74dc4 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák.
new 5ab544a YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan.
new 1aa537c YARN-9397. Fix empty NMResourceInfo object test failures in branch-2
new e1d4d38 YARN-9291. Backport YARN-7637 to branch-2
new 3764197 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda)
new ab3d845 YARN-7143. FileNotFound handling in ResourceUtils is inconsistent
new 8e71ce5 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák.
new f0b57c8 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
new 11f0947 YARN-9271. Backport YARN-6927 for resource type support in MapReduce
new e5d0484 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types
new 9a8b330 YARN-9409. Port resource type changes from YARN-7237 to branch-2
new a8901a1 YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4c6b60a)
\
N -- N -- N refs/heads/YARN-8200 (a8901a1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 51 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/fs/CommonConfigurationKeysPublic.java | 19 +
.../java/org/apache/hadoop/fs/GetSpaceUsed.java | 31 +-
.../org/apache/hadoop/ha/FailoverController.java | 2 +-
.../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++
.../org/apache/hadoop/ha/HAServiceProtocol.java | 18 +
.../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 +
.../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 +
.../HAServiceProtocolClientSideTranslatorPB.java | 16 +
.../HAServiceProtocolServerSideTranslatorPB.java | 20 +
.../java/org/apache/hadoop/http/HttpServer2.java | 15 +
.../main/java/org/apache/hadoop/io/IOUtils.java | 14 +-
.../org/apache/hadoop/io/MultipleIOException.java | 10 +
.../hadoop/io/retry/RetryInvocationHandler.java | 5 +
.../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++
.../org/apache/hadoop/ipc/CallQueueManager.java | 5 +-
.../main/java/org/apache/hadoop/ipc/Client.java | 38 +-
.../org/apache/hadoop/ipc/DecayRpcScheduler.java | 12 +-
.../org/apache/hadoop/ipc/DefaultRpcScheduler.java | 4 +-
.../java/org/apache/hadoop/ipc/ExternalCall.java | 8 +-
.../java/org/apache/hadoop/ipc/IpcException.java | 5 +-
...on.java => ObserverRetryOnActiveException.java} | 21 +-
.../org/apache/hadoop/ipc/ProcessingDetails.java | 96 +++++
.../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 59 ++-
.../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++-
.../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +-
.../java/org/apache/hadoop/ipc/RpcScheduler.java | 11 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 294 +++++++++++--
.../org/apache/hadoop/ipc/WritableRpcEngine.java | 67 ++-
.../hadoop/ipc/metrics/RpcDetailedMetrics.java | 10 +-
.../org/apache/hadoop/ipc/metrics/RpcMetrics.java | 67 ++-
.../org/apache/hadoop/log/LogThrottlingHelper.java | 18 +
.../apache/hadoop/security/LdapGroupsMapping.java | 164 ++++++-
.../java/org/apache/hadoop/util/ProtoUtil.java | 13 +
.../src/main/proto/HAServiceProtocol.proto | 20 +
.../hadoop-common/src/main/proto/RpcHeader.proto | 2 +
.../src/main/resources/core-default.xml | 21 +
.../hadoop-common/src/site/markdown/Metrics.md | 14 +
.../java/org/apache/hadoop/ha/DummyHAService.java | 18 +-
.../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 +
.../java/org/apache/hadoop/io/TestIOUtils.java | 33 ++
.../apache/hadoop/ipc/TestProcessingDetails.java | 61 +++
.../org/apache/hadoop/ipc/TestProtoBufRpc.java | 16 +-
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 26 +-
.../java/org/apache/hadoop/ipc/TestRpcBase.java | 28 ++
.../apache/hadoop/ipc/metrics/TestRpcMetrics.java | 60 +++
.../src/test/proto/test_rpc_service.proto | 1 +
.../hadoop-kms/src/site/markdown/index.md.vm | 4 +-
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 104 +++++
.../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++
.../org/apache/hadoop/hdfs/DFSInputStream.java | 4 +-
.../apache/hadoop/hdfs/NameNodeProxiesClient.java | 54 ++-
.../hadoop/hdfs/protocol/ClientProtocol.java | 58 +++
.../ClientNamenodeProtocolTranslatorPB.java | 37 ++
.../ha/AbstractNNFailoverProxyProvider.java | 19 +-
.../server/namenode/ha/ClientHAProxyFactory.java | 12 +
.../namenode/ha/IPFailoverProxyProvider.java | 2 +-
.../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++
.../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++
.../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++
.../org/apache/hadoop/hdfs/web/JsonUtilClient.java | 55 ++-
.../apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 80 ++--
.../hadoop/hdfs/web/resources/GetOpParam.java | 1 +
.../src/main/proto/ClientNamenodeProtocol.proto | 18 +
.../apache/hadoop/hdfs/protocol/TestReadOnly.java | 95 +++++
.../hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh | 6 +
.../hadoop/fs/http/client/HttpFSFileSystem.java | 86 +++-
.../apache/hadoop/fs/http/server/FSOperations.java | 84 +++-
.../fs/http/server/HttpFSParametersProvider.java | 1 +
.../apache/hadoop/fs/http/server/HttpFSServer.java | 8 +
.../hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh | 10 +-
.../hadoop/fs/http/client/BaseTestHttpFSWith.java | 76 +++-
.../resolver/FederationNamenodeServiceState.java | 3 +
.../federation/router/RouterClientProtocol.java | 15 +
.../server/federation/router/RouterRpcServer.java | 15 +
.../bkjournal/BookKeeperJournalManager.java | 10 +-
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 11 +-
...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 10 +-
.../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 +
.../hdfs/qjournal/client/AsyncLoggerSet.java | 14 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 163 ++++++-
.../hdfs/qjournal/protocol/QJournalProtocol.java | 23 +-
.../QJournalProtocolServerSideTranslatorPB.java | 13 +
.../protocolPB/QJournalProtocolTranslatorPB.java | 17 +
.../hadoop/hdfs/qjournal/server/Journal.java | 93 +++-
.../hdfs/qjournal/server/JournalMetrics.java | 20 +-
.../hadoop/hdfs/qjournal/server/JournalNode.java | 5 +
.../hdfs/qjournal/server/JournalNodeRpcServer.java | 7 +
.../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++
.../hdfs/server/blockmanagement/BlockManager.java | 33 +-
.../blockmanagement/DatanodeAdminManager.java | 147 ++++---
.../hdfs/server/common/HdfsServerConstants.java | 4 +-
.../hdfs/server/common/HttpGetFailedException.java | 25 +-
.../hdfs/server/common/HttpPutFailedException.java | 25 +-
.../org/apache/hadoop/hdfs/server/common/Util.java | 261 +++++++++++-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../hadoop/hdfs/server/datanode/DataStorage.java | 20 +-
.../server/datanode/FSCachingGetSpaceUsed.java | 82 ++++
.../hadoop/hdfs/server/datanode/ReplicaInfo.java | 15 +-
.../server/datanode/fsdataset/FsDatasetSpi.java | 2 +
.../datanode/fsdataset/impl/BlockPoolSlice.java | 11 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 13 +
.../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 108 +++++
.../datanode/web/webhdfs/WebHdfsHandler.java | 3 +-
.../hdfs/server/namenode/BackupJournalManager.java | 2 +-
.../server/namenode/EditLogFileInputStream.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSEditLog.java | 31 +-
.../hdfs/server/namenode/FSEditLogLoader.java | 45 +-
.../hadoop/hdfs/server/namenode/FSImage.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 38 +-
.../hdfs/server/namenode/FSNamesystemLock.java | 66 ++-
.../hdfs/server/namenode/FileJournalManager.java | 9 +-
.../hdfs/server/namenode/GlobalStateIdContext.java | 160 +++++++
.../apache/hadoop/hdfs/server/namenode/INode.java | 13 +
.../hdfs/server/namenode/INodeDirectory.java | 8 +
.../hadoop/hdfs/server/namenode/INodeFile.java | 12 +
.../hadoop/hdfs/server/namenode/ImageServlet.java | 15 +-
.../hadoop/hdfs/server/namenode/JournalSet.java | 11 +-
.../hadoop/hdfs/server/namenode/LogsPurgeable.java | 6 +-
.../server/namenode/NNStorageRetentionManager.java | 2 +-
.../hadoop/hdfs/server/namenode/NameNode.java | 41 +-
.../hdfs/server/namenode/NameNodeHttpServer.java | 8 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 69 ++-
.../namenode/RedundantEditLogInputStream.java | 15 +-
.../hdfs/server/namenode/SecondaryNameNode.java | 2 +-
.../hdfs/server/namenode/TransferFsImage.java | 286 +------------
.../hdfs/server/namenode/ha/EditLogTailer.java | 75 +++-
.../hdfs/server/namenode/ha/StandbyState.java | 19 +-
.../namenode/snapshot/AbstractINodeDiffList.java | 15 +-
.../snapshot/DirectoryWithSnapshotFeature.java | 23 +-
.../namenode/snapshot/FileWithSnapshotFeature.java | 5 +
.../web/resources/NamenodeWebHdfsMethods.java | 14 +-
.../server/protocol/RemoteEditLogManifest.java | 19 +-
.../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 +
.../java/org/apache/hadoop/hdfs/web/JsonUtil.java | 50 ++-
.../hadoop-hdfs/src/main/proto/HdfsServer.proto | 2 +
.../src/main/proto/QJournalProtocol.proto | 17 +
.../src/main/resources/hdfs-default.xml | 46 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../site/markdown/HDFSHighAvailabilityWithQJM.md | 42 +-
.../src/site/markdown/ObserverNameNode.md | 251 +++++++++++
.../hadoop-hdfs/src/site/markdown/WebHDFS.md | 151 +++++++
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +-
.../org/apache/hadoop/hdfs/TestDecommission.java | 156 +++++++
.../hdfs/TestStateAlignmentContextWithHA.java | 350 +++++++++++++++
.../hadoop/hdfs/protocolPB/TestPBHelper.java | 2 +-
.../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +-
.../qjournal/client/TestQuorumJournalManager.java | 161 ++++++-
.../client/TestQuorumJournalManagerUnit.java | 103 ++++-
.../hdfs/qjournal/server/JournalTestUtil.java | 48 +++
.../hadoop/hdfs/qjournal/server/TestJournal.java | 50 ++-
.../qjournal/server/TestJournaledEditsCache.java | 260 ++++++++++++
.../blockmanagement/TestBlockReportLease.java | 170 ++++++++
.../hdfs/server/datanode/SimulatedFSDataset.java | 12 +
.../hdfs/server/datanode/TestDirectoryScanner.java | 68 +++
.../datanode/extdataset/ExternalDatasetImpl.java | 6 +
.../impl/TestReplicaCachingGetSpaceUsed.java | 148 +++++++
.../server/namenode/NNThroughputBenchmark.java | 6 +-
.../hdfs/server/namenode/NameNodeAdapter.java | 13 +
.../hadoop/hdfs/server/namenode/TestEditLog.java | 18 +-
.../namenode/TestEditLogFileInputStream.java | 18 +
.../hdfs/server/namenode/TestFSEditLogLoader.java | 47 +++
.../server/namenode/TestFSImageWithSnapshot.java | 100 ++++-
.../server/namenode/TestFileJournalManager.java | 2 +-
.../hadoop/hdfs/server/namenode/TestFsck.java | 2 +-
.../server/namenode/TestGenericJournalConf.java | 2 +-
.../namenode/TestNNStorageRetentionManager.java | 4 +-
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 146 ++++++-
.../namenode/ha/TestConsistentReadsObserver.java | 415 ++++++++++++++++++
.../namenode/ha/TestDelegationTokensWithHA.java | 50 +++
.../hdfs/server/namenode/ha/TestEditLogTailer.java | 144 ++++++-
.../server/namenode/ha/TestFailureToReadEdits.java | 19 +-
.../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++
.../hdfs/server/namenode/ha/TestObserverNode.java | 379 +++++++++++++++++
.../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++
.../namenode/ha/TestStandbyInProgressTail.java | 468 ++++++++++++++++++++
.../namenode/snapshot/SnapshotTestHelper.java | 3 +-
.../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +-
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++
.../org/apache/hadoop/hdfs/web/TestWebHDFS.java | 94 +++++
.../org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java | 37 +-
.../test/java/org/apache/hadoop/hdfs/NNBench.java | 2 +-
.../hadoop/mapreduce/v2/MiniMRYarnCluster.java | 4 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 6 +
.../src/main/resources/yarn-default.xml | 9 +
.../yarn/server/nodemanager/NodeManager.java | 2 +
.../containermanager/ContainerManagerImpl.java | 51 ++-
.../launcher/AbstractContainersLauncher.java | 41 ++
.../launcher/ContainersLauncher.java | 35 +-
.../containermanager/launcher/package-info.java | 18 +-
.../container-executor/impl/container-executor.c | 73 +++-
.../test/test-container-executor.c | 12 +
.../yarn/server/nodemanager/TestNodeManager.java | 28 ++
.../yarn/server/resourcemanager/AdminService.java | 7 +
.../server/resourcemanager/ResourceManager.java | 1 +
.../scheduler/SchedulerApplicationAttempt.java | 5 +-
.../scheduler/capacity/CSQueueUtils.java | 61 +--
.../capacity/CapacitySchedulerConfiguration.java | 5 +-
.../scheduler/common/fica/FiCaSchedulerApp.java | 4 +-
.../yarn/server/resourcemanager/TestRMRestart.java | 5 +-
.../resourcemanager/TestResourceManager.java | 30 +-
.../capacity/TestContainerAllocation.java | 4 +
.../scheduler/capacity/TestLeafQueue.java | 9 +
.../capacity/TestNodeLabelContainerAllocation.java | 165 +++++++-
.../src/site/markdown/CapacityScheduler.md | 2 +-
207 files changed, 10139 insertions(+), 992 deletions(-)
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java
copy hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/{IpcException.java => ObserverRetryOnActiveException.java} (62%)
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProcessingDetails.java
create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProcessingDetails.java
create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/metrics/TestRpcMetrics.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java
copy hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java => hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HttpGetFailedException.java (61%)
copy hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java => hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HttpPutFailedException.java (64%)
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSCachingGetSpaceUsed.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaCachingGetSpaceUsed.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestReplicaCachingGetSpaceUsed.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyInProgressTail.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/AbstractContainersLauncher.java
copy hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/IpcException.java => hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/package-info.java (70%)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org