You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by el...@apache.org on 2019/10/04 12:22:26 UTC
[hadoop] branch HDDS-1880-Decom updated (fd5e877 -> ec70207)
This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from fd5e877 Merge branch 'trunk' into HDDS-1880-Decom
add 3d78b12 YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar
add 3fd3d74 HDDS-2161. Create RepeatedKeyInfo structure to be saved in deletedTable
add 6cbe5d3 HDDS-2160. Add acceptance test for ozonesecure-mr compose. Contributed by Xiaoyu Yao. (#1490)
add 0a716bd HDDS-2159. Fix Race condition in ProfileServlet#pid.
add bfe1dac HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496)
add e8e7d7b HADOOP-16561. [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes (#1500)
add 8f1a135 HDDS-2081. Fix TestRatisPipelineProvider#testCreatePipelinesDnExclude. Contributed by Aravindan Vijayan. (#1506)
add 51c64b3 HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running
add 91f50b9 HDDS-2167. Hadoop31-mr acceptance test is failing due to the shading
add 43203b4 HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun.
add 816d3cb HDFS-14837. Review of Block.java. Contributed by David Mollitor.
add afa1006 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor.
add f16cf87 HDDS-2170. Add Object IDs and Update ID to Volume Object (#1510)
add eb96a30 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena.
add 66400c1 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena.
add c2731d4 YARN-9730. Support forcing configured partitions to be exclusive based on app node label
add 6917754 HDDS-2172.Ozone shell should remove description about REST protocol support. Contributed by Siddharth Wagle.
add a346381 HDDS-2168. TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory error (#1509)
add 3f89084 HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs.
add bec0864 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal
add c724577 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko
add 8baebb5 HDDS-2171. Dangling links in test report due to incompatible realpath
add e6fb6ee HDDS-1738. Add nullable annotation for OMResponse classes
add e346e36 HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
add 16f626f HDDS-2165. Freon fails if bucket does not exists
add c89d22d HADOOP-16602. mvn package fails in hadoop-aws.
add bdaaa3b HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina
add f647185 HDDS-2067. Create generic service facade with tracing/metrics/logging support
add 606e341 Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label
add 587a8ee HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir (#1522). Contributed by Gabor Bota.
add 7b6219a HDDS-2182. Fix checkstyle violations introduced by HDDS-1738
add a3f6893 HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui.
add 1a2a352 HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya.
add 18a8c24 YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein
add 06998a1 HDDS-2180. Add Object ID and update ID on VolumeList Object. (#1526)
add b1e55cf HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He.
add 2adcc3c HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang.
add c55ac6a HDDS-2174. Delete GDPR Encryption Key from metadata when a Key is deleted
add b6ef8cc HDD-2193. Adding container related metrics in SCM.
add 0371e95 HDDS-2179. ConfigFileGenerator fails with Java 10 or newer
add 9bf7a6e HDDS-2149. Replace findbugs with spotbugs
add 2870668 Make upstream aware of 3.1.3 release.
add 8a9ede5 HADOOP-15616. Incorporate Tencent Cloud COS File System Implementation. Contributed by Yang Yu.
add a93a139 HDDS-2185. createmrenv failure not reflected in acceptance test result
add ce58c05 HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
add 13b427f HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar.
add 14b4fbc HDDS-1146. Adding container related metrics in SCM. (#1541)
add 0d5d0b9 Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao."
add c4c8d5f HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
add 2200871 HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun.
add d8313b2 HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun.
add c0edc84 HADOOP-16548 : Disable Flush() over config
add 760b523 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao."
add d6b0a8d HDDS-2183. Container and pipline subcommands of scmcli should be grouped
add a530ac3 HDDS-2153. Add a config to tune max pending requests in Ratis leader
add b46d823 HDDS-2202. Remove unused import in OmUtils
add 98ca07e HDDS-2207. Update Ratis to latest snapshot. Contributed by Shashikant Bannerjee. (#1550)
add 4d3c580 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi.
add e5bba59 HDDS-2205. checkstyle.sh reports wrong failure count
add b3275ab HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko.
add 137546a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph
add 8efd25b HDDS-1615. ManagedChannel references are being leaked in ReplicationSupervisor.java. Contributed by Mukul Kumar Singh. (#1547)
add 425a6c8 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung.
add 99cd757 YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph
add 6ef6594 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370)
add 918b470 HDDS-2166. Some RPC metrics are missing from SCM prometheus endpoint
add 1921e94 HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3
add d947ded HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10 (#1554). Contributed by Siyao Meng.
add 770adc5 HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists
add d1ddba6 YARN-9862. Increase yarn-services-core test timeout value. Contributed by Prabhu Joseph
add 3df733c HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li.
add 41440ec HDDS-2210. ContainerStateMachine should not be marked unhealthy if applyTransaction fails with closed container exception(#1552).
add f1ba9bf HDDS-2187. ozone-mr test fails with No FileSystem for scheme "o3fs"
add 61a8436 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi.
add 2e1fd44 HDDS-2201. Rename VolumeList to UserVolumeInfo. (#1566)
add 0d2d6f9 YARN-9792. Document examples of SchedulerConf with Node Labels. Contributed by Prabhu Joseph
add e8ae632 HDDS-2068. Make StorageContainerDatanodeProtocolService message based
add ffd4e52 HDDS-2073. Make SCMSecurityProtocol message based. Contributed by Elek, Marton.
add 685918e HDDS-2227. GDPR key generation could benefit from secureRandom. (#1574)
add 169cef7 HDDS-2162. Make OM Generic related configuration support HA style config. (#1511)
add 53ed78b HDDS-2224. Fix loadup cache for cache cleanup policy NEVER. (#1567)
add b09d389 HDDS-2019. Handle Set DtService of token in S3Gateway for OM HA. (#1489)
add 559ee27 HADOOP-16599. Allow a SignerInitializer to be specified along with a Custom Signer
add 1303255 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang.
add 4c24f24 HDDS-2072. Make StorageContainerLocationProtocolService message based Contributed by Elek, Marton.
add c5665b2 HDDS-2228. Fix NPE in OzoneDelegationTokenManager#addPersistedDelegat… (#1571)
add 0e026cb HADOOP-16620. [pb-upgrade] Remove protocol buffers 3.7.1 from requirements in BUILDING.txt (#1558)
add c19fa3d HADOOP-16605. Fix testcase testSSLChannelModeConfig
add 5a7483c HDFS-14888. RBF: Enable Parallel Test Profile for builds. Contributed by Ayush Saxena.
add d59bcbf HDDS-2226. S3 Secrets should use a strong RNG. (#1572)
add a3fe404 HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C.
add 51eaeca HDDS-2211. Collect docker logs if env fails to start (#1553)
add 47d721d HDDS-2234. rat.sh fails due to ozone-recon-web/build files (#1580)
add 9446686 HDDS-2231. test-single.sh cannot copy results (#1575)
add 76605f1 HDDS-1720 : Add ability to configure RocksDB logs for Ozone Manager.
add b7cb8fe HDDS-2200 : Recon does not handle the NULL snapshot from OM DB cleanly.
add 1dde3ef HADOOP-16624. Upgrade hugo to the latest version in Dockerfile
add cdaa480 HDDS-2198. SCM should not consider containers in CLOSING state to come out of safemode. (#1540)
add 9700e20 HDDS-2223. Support ReadWrite lock in LockManager. (#1564)
add 844b766 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)"
add c99a121 HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell.
add ec8f691 HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config
add b23bdaf HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina
add 2478cba YARN-9782. Avoid DNS resolution while running SLS. Contributed by Abhishek Modi.
add 4cf0b36 HDDS-2222 (#1578)
add a9849f6 Revert "HDDS-2222 (#1578)" (#1594)
add bffcd33 HDDS-2230. Invalid entries in ozonesecure-mr config
add d061c84 HDDS-2140. Add robot test for GDPR feature
add 6171a41 HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host
new ec70207 Merge remote-tracking branch 'origin/trunk' into HDDS-1880-Decom
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
BUILDING.txt | 31 -
dev-support/docker/Dockerfile | 19 +-
.../hadoop-cos/dev-support/findbugs-exclude.xml | 0
hadoop-cloud-storage-project/hadoop-cos/pom.xml | 140 ++++
.../site/markdown/cloud-storage/index.md | 367 ++++++++++
.../hadoop-cos}/site/resources/css/site.css | 0
.../java/org/apache/hadoop/fs/cosn/BufferPool.java | 245 +++++++
.../hadoop/fs/cosn/ByteBufferInputStream.java | 89 +++
.../hadoop/fs/cosn/ByteBufferOutputStream.java | 74 ++
.../apache/hadoop/fs/cosn/ByteBufferWrapper.java | 103 +++
.../java/org/apache/hadoop/fs/cosn/Constants.java | 35 +-
.../main/java/org/apache/hadoop/fs/cosn/CosN.java | 31 +-
.../org/apache/hadoop/fs/cosn/CosNConfigKeys.java | 86 +++
.../apache/hadoop/fs/cosn/CosNCopyFileContext.java | 66 ++
.../apache/hadoop/fs/cosn/CosNCopyFileTask.java | 68 ++
.../apache/hadoop/fs/cosn/CosNFileReadTask.java | 125 ++++
.../org/apache/hadoop/fs/cosn/CosNFileSystem.java | 814 +++++++++++++++++++++
.../org/apache/hadoop/fs/cosn/CosNInputStream.java | 365 +++++++++
.../apache/hadoop/fs/cosn/CosNOutputStream.java | 284 +++++++
.../java/org/apache/hadoop/fs/cosn/CosNUtils.java | 167 +++++
.../hadoop/fs/cosn/CosNativeFileSystemStore.java | 768 +++++++++++++++++++
.../org/apache/hadoop/fs/cosn/FileMetadata.java | 68 ++
.../hadoop/fs/cosn/NativeFileSystemStore.java | 99 +++
.../org/apache/hadoop/fs/cosn/PartialListing.java | 64 ++
.../main/java/org/apache/hadoop/fs/cosn/Unit.java | 27 +-
.../fs/cosn/auth/COSCredentialProviderList.java | 139 ++++
.../EnvironmentVariableCredentialProvider.java | 55 ++
.../fs/cosn/auth/NoAuthWithCOSException.java | 32 +-
.../fs/cosn/auth/SimpleCredentialProvider.java | 54 ++
.../apache/hadoop/fs/cosn/auth/package-info.java | 19 +-
.../org/apache/hadoop/fs/cosn/package-info.java | 19 +-
.../apache/hadoop/fs/cosn/CosNTestConfigKey.java | 30 +-
.../org/apache/hadoop/fs/cosn/CosNTestUtils.java | 78 ++
.../apache/hadoop/fs/cosn/TestCosNInputStream.java | 167 +++++
.../hadoop/fs/cosn/TestCosNOutputStream.java | 87 +++
.../hadoop/fs/cosn/contract/CosNContract.java | 36 +-
.../fs/cosn/contract/TestCosNContractCreate.java | 26 +-
.../fs/cosn/contract/TestCosNContractDelete.java | 26 +-
.../fs/cosn/contract/TestCosNContractDistCp.java | 54 ++
.../contract/TestCosNContractGetFileStatus.java | 27 +-
.../fs/cosn/contract/TestCosNContractMkdir.java | 26 +-
.../fs/cosn/contract/TestCosNContractOpen.java | 26 +-
.../fs/cosn/contract/TestCosNContractRename.java | 26 +-
.../fs/cosn/contract/TestCosNContractRootDir.java | 27 +-
.../fs/cosn/contract/TestCosNContractSeek.java | 26 +-
.../hadoop/fs/cosn/contract/package-info.java | 19 +-
.../src/test/resources/contract/cosn.xml | 120 +++
.../hadoop-cos/src/test/resources/core-site.xml | 107 +++
.../hadoop-cos/src/test/resources/log4j.properties | 18 +
hadoop-cloud-storage-project/pom.xml | 1 +
.../apache/hadoop/crypto/CryptoInputStream.java | 67 +-
.../org/apache/hadoop/fs/AbstractFileSystem.java | 16 +-
.../hadoop/fs/ByteBufferPositionedReadable.java | 24 +
.../org/apache/hadoop/fs/ChecksumFileSystem.java | 22 +
.../apache/hadoop/fs/CommonPathCapabilities.java | 126 ++++
.../org/apache/hadoop/fs/DelegateToFileSystem.java | 7 +
.../org/apache/hadoop/fs/FSDataInputStream.java | 23 +-
.../java/org/apache/hadoop/fs/FileContext.java | 23 +-
.../main/java/org/apache/hadoop/fs/FileSystem.java | 30 +-
.../org/apache/hadoop/fs/FilterFileSystem.java | 7 +
.../main/java/org/apache/hadoop/fs/FilterFs.java | 5 +
.../main/java/org/apache/hadoop/fs/Globber.java | 208 +++++-
.../java/org/apache/hadoop/fs/HarFileSystem.java | 19 +-
.../org/apache/hadoop/fs/PathCapabilities.java | 61 ++
.../org/apache/hadoop/fs/RawLocalFileSystem.java | 19 +
.../hadoop/fs/http/AbstractHttpFileSystem.java | 18 +
.../apache/hadoop/fs/impl/FsLinkResolution.java | 98 +++
.../hadoop/fs/impl/PathCapabilitiesSupport.java | 40 +-
.../java/org/apache/hadoop/fs/shell/Mkdir.java | 4 +-
.../hadoop/fs/viewfs/ChRootedFileSystem.java | 6 +
.../apache/hadoop/fs/viewfs/ViewFileSystem.java | 32 +
.../apache/hadoop/util/NodeHealthScriptRunner.java | 1 +
.../src/site/markdown/DeprecatedProperties.md | 4 +
.../src/site/markdown/filesystem/filesystem.md | 5 +-
.../src/site/markdown/filesystem/index.md | 1 +
.../site/markdown/filesystem/pathcapabilities.md | 158 ++++
.../site/markdown/release/3.1.3/CHANGES.3.1.3.md | 336 +++++++++
.../markdown/release/3.1.3/RELEASENOTES.3.1.3.md | 59 ++
.../hadoop/crypto/CryptoStreamsTestBase.java | 115 ++-
.../apache/hadoop/crypto/TestCryptoStreams.java | 24 +
.../hadoop/crypto/TestCryptoStreamsForLocalFS.java | 5 +
.../hadoop/crypto/TestCryptoStreamsNormal.java | 5 +
.../fs/contract/AbstractContractAppendTest.java | 10 +
.../fs/contract/AbstractContractConcatTest.java | 7 +
.../hadoop/fs/contract/ContractTestUtils.java | 50 +-
.../org/apache/hadoop/test/LambdaTestUtils.java | 3 +
.../hadoop/util/TestNodeHealthScriptRunner.java | 9 +
hadoop-hdds/common/pom.xml | 4 +-
.../function/FunctionWithServiceException.java | 29 +-
.../apache/hadoop/hdds/function/package-info.java | 15 +-
.../SCMSecurityProtocolClientSideTranslatorPB.java | 104 ++-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 132 ----
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
...inerLocationProtocolClientSideTranslatorPB.java | 411 +++++------
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 24 +-
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 62 ++
.../apache/hadoop/hdds/utils/db/TypedTable.java | 2 +-
.../hadoop/hdds/utils/db/cache/TableCache.java | 9 +
.../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 7 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../org/apache/hadoop/ozone/lock/ActiveLock.java | 63 +-
.../org/apache/hadoop/ozone/lock/LockManager.java | 166 ++++-
...inerLocationProtocolServerSideTranslatorPB.java | 389 ----------
.../src/main/proto/SCMSecurityProtocol.proto | 96 ++-
.../src/main/proto/ScmBlockLocationProtocol.proto | 2 +-
.../proto/StorageContainerLocationProtocol.proto | 185 +++--
.../common/src/main/resources/ozone-default.xml | 8 +
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 16 +-
.../apache/hadoop/ozone/lock/TestLockManager.java | 145 +++-
.../hadoop/hdds/conf/ConfigFileGenerator.java | 5 +-
hadoop-hdds/container-service/pom.xml | 9 +-
.../server/ratis/ContainerStateMachine.java | 20 +-
.../transport/server/ratis/XceiverServerRatis.java | 8 +-
.../replication/GrpcReplicationClient.java | 6 +
...inerDatanodeProtocolClientSideTranslatorPB.java | 60 +-
...inerDatanodeProtocolServerSideTranslatorPB.java | 115 +--
.../proto/StorageContainerDatanodeProtocol.proto | 58 +-
.../ozone/container/common/SCMTestUtils.java | 4 +-
.../server/OzoneProtocolMessageDispatcher.java | 88 +++
.../apache/hadoop/hdds/server/ProfileServlet.java | 10 +-
.../hadoop/hdds/server/PrometheusMetricsSink.java | 16 +-
.../hdds/server/TestPrometheusMetricsSink.java | 77 +-
hadoop-hdds/pom.xml | 17 +-
hadoop-hdds/server-scm/pom.xml | 4 +-
.../container/AbstractContainerReportHandler.java | 8 +
.../hdds/scm/container/ContainerManager.java | 8 +
.../hdds/scm/container/ContainerReportHandler.java | 2 +
.../IncrementalContainerReportHandler.java | 10 +
.../hdds/scm/container/SCMContainerManager.java | 48 +-
.../metrics/SCMContainerManagerMetrics.java | 144 ++++
.../hdds/scm/container/metrics/package-info.java | 22 +
.../apache/hadoop/hdds/scm/node/NodeManager.java | 8 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 51 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 186 +++++
...lockLocationProtocolServerSideTranslatorPB.java | 56 +-
...inerLocationProtocolServerSideTranslatorPB.java | 393 ++++++++++
.../hadoop/hdds/scm/protocol/package-info.java | 22 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 26 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 9 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 15 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 102 ++-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 27 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 36 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 67 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 6 +-
.../scm/server/TestSCMBlockProtocolServer.java | 3 +-
.../testutils/ReplicationNodeManagerMock.java | 5 +-
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 22 +-
.../hdds/scm/cli/container/CloseSubcommand.java | 7 +-
...CloseSubcommand.java => ContainerCommands.java} | 37 +-
.../hdds/scm/cli/container/CreateSubcommand.java | 5 +-
.../hdds/scm/cli/container/DeleteSubcommand.java | 7 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 5 +-
.../hdds/scm/cli/container/ListSubcommand.java | 5 +-
.../cli/pipeline/ActivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ClosePipelineSubcommand.java | 11 +-
.../cli/pipeline/DeactivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ListPipelinesSubcommand.java | 11 +-
.../PipelineCommands.java} | 38 +-
.../org/apache/hadoop/hdfs/DFSInputStream.java | 14 +
.../apache/hadoop/hdfs/DistributedFileSystem.java | 22 +
.../hadoop/hdfs/client/DfsPathCapabilities.java | 62 ++
.../org/apache/hadoop/hdfs/protocol/Block.java | 141 +++-
.../namenode/ha/ObserverReadProxyProvider.java | 26 +-
.../apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 30 +
.../hadoop/hdfs/TestDefaultNameNodePort.java | 38 +-
.../hadoop/fs/http/client/HttpFSFileSystem.java | 30 +
.../fs/http/server/HttpFSAuthenticationFilter.java | 16 +-
.../fs/http/server/HttpFSServerWebServer.java | 22 +
.../service/hadoop/FileSystemAccessService.java | 21 +-
.../src/main/resources/httpfs-default.xml | 20 +-
.../main/native/libhdfs-tests/test_libhdfs_ops.c | 43 +-
.../src/main/native/libhdfs/hdfs.c | 117 +++
.../src/main/native/libhdfs/include/hdfs/hdfs.h | 21 +-
.../src/main/native/libhdfspp/tests/hdfs_shim.c | 6 +
.../libhdfspp/tests/libhdfs_wrapper_defines.h | 1 +
.../libhdfspp/tests/libhdfs_wrapper_undefs.h | 1 +
.../libhdfspp/tests/libhdfspp_wrapper_defines.h | 1 +
hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 48 ++
.../src/main/webapps/router/federationhealth.html | 10 +-
.../src/main/webapps/router/federationhealth.js | 7 +-
.../src/main/webapps/static/rbf.css | 26 +-
.../fs/contract/router/RouterHDFSContract.java | 17 +-
.../fs/contract/router/SecurityConfUtil.java | 15 +-
.../router/TestRouterHDFSContractAppendSecure.java | 4 +-
.../router/TestRouterHDFSContractConcatSecure.java | 5 +-
.../router/TestRouterHDFSContractCreateSecure.java | 5 +-
.../TestRouterHDFSContractDelegationToken.java | 2 +-
.../router/TestRouterHDFSContractDeleteSecure.java | 4 +-
.../TestRouterHDFSContractGetFileStatusSecure.java | 5 +-
.../router/TestRouterHDFSContractMkdirSecure.java | 5 +-
.../router/TestRouterHDFSContractOpenSecure.java | 5 +-
.../router/TestRouterHDFSContractRenameSecure.java | 5 +-
.../TestRouterHDFSContractRootDirectorySecure.java | 5 +-
.../router/TestRouterHDFSContractSeekSecure.java | 5 +-
.../TestRouterHDFSContractSetTimesSecure.java | 5 +-
.../server/federation/router/TestRouterQuota.java | 2 +-
.../security/TestRouterHttpDelegationToken.java | 1 +
.../dev-support/jdiff/Apache_Hadoop_HDFS_3.1.3.xml | 676 +++++++++++++++++
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 11 +-
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +-
.../hadoop/hdfs/qjournal/client/QuorumCall.java | 18 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 2 +
.../token/block/BlockTokenSecretManager.java | 31 +-
.../hdfs/server/blockmanagement/BlockInfo.java | 6 +
.../blockmanagement/BlockInfoContiguous.java | 14 +
.../server/blockmanagement/BlockInfoStriped.java | 9 +
.../hdfs/server/blockmanagement/BlockManager.java | 94 ++-
.../blockmanagement/BlockPlacementStatus.java | 8 +
.../BlockPlacementStatusDefault.java | 8 +
.../BlockPlacementStatusWithNodeGroup.java | 11 +
.../BlockPlacementStatusWithUpgradeDomain.java | 22 +-
.../blockmanagement/BlockReportLeaseManager.java | 8 +-
.../hdfs/server/namenode/INodeDirectory.java | 6 +
.../hadoop/hdfs/server/namenode/INodeFile.java | 3 +
.../hdfs/server/namenode/NameNodeRpcServer.java | 14 +-
.../namenode/snapshot/AbstractINodeDiffList.java | 4 +
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 2 +-
.../src/main/resources/hdfs-default.xml | 19 +
.../src/main/webapps/hdfs/dfshealth.html | 4 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 4 +-
.../src/site/markdown/ObserverNameNode.md | 14 +
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 17 +-
.../hadoop/hdfs/TestBlockMissingException.java | 2 -
.../apache/hadoop/hdfs/TestByteBufferPread.java | 27 +-
.../java/org/apache/hadoop/hdfs/TestClose.java | 1 -
.../hdfs/TestStateAlignmentContextWithHA.java | 1 +
.../hadoop/hdfs/qjournal/MiniJournalCluster.java | 6 +-
.../qjournal/client/TestQuorumJournalManager.java | 34 +-
.../hdfs/security/token/block/TestBlockToken.java | 23 +
.../blockmanagement/BlockManagerTestUtil.java | 30 +-
.../hdfs/server/blockmanagement/TestBlockInfo.java | 36 +
.../server/blockmanagement/TestBlockManager.java | 58 ++
.../TestBlockPlacementStatusDefault.java | 57 ++
.../TestBlockPlacementStatusWithUpgradeDomain.java | 59 +-
.../TestBlocksWithNotEnoughRacks.java | 111 ++-
.../namenode/ha/TestConsistentReadsObserver.java | 2 +
.../ha/TestFailoverWithBlockTokensEnabled.java | 36 +-
.../server/namenode/ha/TestMultiObserverNode.java | 2 +
.../hdfs/server/namenode/ha/TestObserverNode.java | 2 +
.../namenode/snapshot/TestRenameWithSnapshots.java | 15 +-
.../namenode/snapshot/TestSnapshotDeletion.java | 2 +-
.../hadoop-mapreduce-client-common/pom.xml | 34 +-
.../src/main/proto/HSAdminRefreshProtocol.proto | 3 +-
.../src/main/proto/MRClientProtocol.proto | 1 +
.../src/main/proto/mr_protos.proto | 1 +
.../src/main/proto/mr_service_protos.proto | 1 +
.../org/apache/hadoop/mapred/FileInputFormat.java | 5 +-
.../hadoop/mapred/InvalidInputException.java | 4 +
.../hadoop/mapred/LocatedFileStatusFetcher.java | 66 +-
.../mapreduce/lib/input/FileInputFormat.java | 6 +-
.../mapreduce/lib/input/InvalidInputException.java | 4 +
.../hadoop-mapreduce-client-shuffle/pom.xml | 27 +-
.../src/main/proto/ShuffleHandlerRecovery.proto | 1 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 3 +-
hadoop-ozone/common/pom.xml | 4 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 57 +-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 8 +-
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 52 ++
...lumeListCodec.java => UserVolumeInfoCodec.java} | 13 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 100 ++-
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 91 +++
.../hadoop/ozone/security/GDPRSymmetricKey.java | 8 +-
.../OzoneDelegationTokenSecretManager.java | 5 +-
.../hadoop/ozone/security/OzoneSecretManager.java | 7 +-
.../src/main/proto/OzoneManagerProtocol.proto | 18 +-
.../ozone/security/TestGDPRSymmetricKey.java | 4 +-
hadoop-ozone/csi/pom.xml | 4 +-
.../dev-support/checks/_mvn_unit_report.sh | 16 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 2 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 12 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 39 +-
.../src/main/compose/ozonesecure-mr/docker-config | 31 +-
.../{test-single.sh => ozonesecure-mr/test.sh} | 37 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 4 +-
hadoop-ozone/dist/src/main/compose/test-single.sh | 2 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 27 +-
.../dist/src/main/smoketest/gdpr/gdpr.robot | 89 +++
.../smoketest/{kinit.robot => kinit-hadoop.robot} | 2 +-
hadoop-ozone/dist/src/main/smoketest/kinit.robot | 5 +-
.../dist/src/main/smoketest/mapreduce.robot | 2 +-
hadoop-ozone/insight/pom.xml | 9 +-
.../ozone/insight/BaseInsightSubCommand.java | 7 +-
.../scm/ScmProtocolBlockLocationInsight.java | 6 +-
...va => ScmProtocolContainerLocationInsight.java} | 18 +-
...nsight.java => ScmProtocolDatanodeInsight.java} | 27 +-
...nsight.java => ScmProtocolSecurityInsight.java} | 18 +-
.../metrics/TestSCMContainerManagerMetrics.java | 167 +++++
.../scm/pipeline/TestRatisPipelineProvider.java | 26 +-
.../rpc/TestContainerStateMachineFailures.java | 65 ++
.../client/rpc/TestOzoneRpcClientAbstract.java | 76 +-
.../ozone/om/TestOzoneManagerConfiguration.java | 9 +-
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 97 +++
hadoop-ozone/ozone-manager/pom.xml | 5 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 40 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 113 +--
.../org/apache/hadoop/ozone/om/OzoneManager.java | 232 +-----
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 35 +-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 306 ++++++++
.../hadoop/ozone/om/{ => ha}/OMNodeDetails.java | 2 +-
.../apache/hadoop/ozone/om/ha/package-info.java | 14 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 3 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 8 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 4 +-
.../om/request/volume/OMVolumeCreateRequest.java | 12 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 5 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 32 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 10 +-
.../om/response/bucket/OMBucketCreateResponse.java | 8 +-
.../om/response/bucket/OMBucketDeleteResponse.java | 4 +-
.../bucket/OMBucketSetPropertyResponse.java | 7 +-
.../response/file/OMDirectoryCreateResponse.java | 3 +-
.../om/response/file/OMFileCreateResponse.java | 3 +-
.../om/response/key/OMAllocateBlockResponse.java | 6 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 7 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 3 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 32 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 4 +-
.../ozone/om/response/key/OMKeyRenameResponse.java | 6 +-
.../multipart/S3MultipartUploadAbortResponse.java | 21 +-
.../S3MultipartUploadCommitPartResponse.java | 45 +-
.../S3MultipartUploadCompleteResponse.java | 3 +-
.../om/response/volume/OMVolumeAclOpResponse.java | 3 +-
.../om/response/volume/OMVolumeCreateResponse.java | 13 +-
.../om/response/volume/OMVolumeDeleteResponse.java | 10 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 12 +-
.../response/volume/OMVolumeSetQuotaResponse.java | 4 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 2 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 41 +-
.../org/apache/hadoop/ozone/web/ozShell/Shell.java | 3 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 21 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/TestOMRequestUtils.java | 25 +-
.../hadoop/ozone/om/request/package-info.java} | 18 +-
.../request/volume/TestOMVolumeCreateRequest.java | 16 +-
.../volume/TestOMVolumeSetOwnerRequest.java | 4 +-
.../ozone/om/response/TestOMResponseUtils.java | 8 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 20 +-
.../hadoop/ozone/om/response/package-info.java} | 18 +-
.../s3/multipart/TestS3MultipartResponse.java | 3 +-
.../TestS3MultipartUploadAbortResponse.java | 19 +-
.../volume/TestOMVolumeCreateResponse.java | 7 +-
.../volume/TestOMVolumeDeleteResponse.java | 11 +-
.../volume/TestOMVolumeSetOwnerResponse.java | 17 +-
.../ozone/om/response/volume/package-info.java | 22 +-
.../TestOzoneDelegationTokenSecretManager.java | 29 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 4 +-
hadoop-ozone/ozonefs-lib-legacy/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 9 +-
hadoop-ozone/pom.xml | 14 +-
hadoop-ozone/recon/pom.xml | 4 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 35 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 21 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 28 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 33 +-
.../recon/spi/impl/ReconContainerDBProvider.java | 32 +-
.../ozone/recon/AbstractOMMetadataManagerTest.java | 2 +-
.../apache/hadoop/ozone/recon/TestReconUtils.java | 31 +
.../recovery/TestReconOmMetadataManagerImpl.java | 133 ++--
.../impl/TestOzoneManagerServiceProviderImpl.java | 29 +-
.../spi/impl/TestReconContainerDBProvider.java | 13 -
hadoop-ozone/s3gateway/pom.xml | 5 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 11 +-
.../hadoop/ozone/s3/OzoneServiceProvider.java | 50 +-
.../apache/hadoop/ozone/s3/util/OzoneS3Util.java | 44 ++
.../hadoop/ozone/s3/util/TestOzoneS3Util.java | 130 ++++
hadoop-ozone/tools/pom.xml | 9 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 3 +-
.../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 4 +-
.../services/org.apache.hadoop.fs.FileSystem | 0
hadoop-ozone/upgrade/pom.xml | 5 +-
hadoop-project/pom.xml | 4 +-
hadoop-project/src/site/site.xml | 1 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 28 +-
.../java/org/apache/hadoop/fs/s3a/Invoker.java | 3 +-
.../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 6 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 76 +-
.../hadoop/fs/s3a/auth/AwsSignerInitializer.java | 53 ++
.../hadoop/fs/s3a/{ => auth}/SignerManager.java | 80 +-
.../auth/delegation/DelegationTokenProvider.java | 20 +-
.../hadoop/fs/s3a/commit/CommitConstants.java | 22 +-
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 3 +-
.../hadoop/fs/s3a/select/SelectConstants.java | 2 +-
.../apache/hadoop/fs/s3a/select/SelectTool.java | 2 +-
.../src/site/markdown/tools/hadoop-aws/index.md | 58 ++
.../fs/s3a/ITestLocatedFileStatusFetcher.java | 31 +-
.../hadoop/fs/s3a/ITestS3AFSMainOperations.java | 65 ++
.../hadoop/fs/s3a/ITestS3AMiscOperations.java | 7 +
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 7 +-
.../apache/hadoop/fs/s3a/TestSignerManager.java | 130 ----
.../hadoop/fs/s3a/auth/ITestCustomSigner.java | 237 ++++++
.../fs/s3a/auth/ITestRestrictedReadAccess.java | 707 ++++++++++++++++++
.../hadoop/fs/s3a/auth/TestSignerManager.java | 590 +++++++++++++++
.../fs/s3a/commit/ITestCommitOperations.java | 5 +-
.../s3a/s3guard/AbstractS3GuardToolTestBase.java | 2 +-
.../apache/hadoop/fs/s3a/select/ITestS3Select.java | 4 +-
.../org/apache/hadoop/fs/adl/AdlFileSystem.java | 18 +
.../fs/adl/live/TestAdlSdkConfiguration.java | 1 +
.../hadoop/fs/azure/NativeAzureFileSystem.java | 17 +
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 14 +
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 22 +-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 6 +-
.../fs/azurebfs/constants/ConfigurationKeys.java | 8 +
.../constants/FileSystemConfigurations.java | 1 +
.../fs/azurebfs/services/AbfsOutputStream.java | 7 +-
.../hadoop-azure/src/site/markdown/abfs.md | 17 +
.../fs/azurebfs/ITestAzureBlobFileSystemFlush.java | 27 +-
.../fs/azurebfs/ITestGetNameSpaceEnabled.java | 6 +-
.../org/apache/hadoop/tools/DistCpConstants.java | 6 +
.../apache/hadoop/tools/mapred/CopyCommitter.java | 10 +-
.../org/apache/hadoop/tools/mapred/CopyMapper.java | 3 +-
.../tools/mapred/RetriableFileCopyCommand.java | 34 +-
.../org/apache/hadoop/tools/util/DistCpUtils.java | 26 +-
.../hadoop/tools/mapred/TestCopyCommitter.java | 5 +-
.../apache/hadoop/tools/mapred/TestCopyMapper.java | 56 ++
.../tools/mapred/TestRetriableFileCopyCommand.java | 25 +-
.../apache/hadoop/tools/util/TestDistCpUtils.java | 45 +-
.../tools/util/TestDistCpUtilsWithCombineMode.java | 4 +-
.../BlockPlacementPolicyAlwaysSatisfied.java | 5 +
.../java/org/apache/hadoop/yarn/sls/SLSRunner.java | 25 +
.../hadoop/yarn/sls/conf/SLSConfiguration.java | 3 +
.../apache/hadoop/yarn/sls/BaseSLSRunnerTest.java | 4 +-
.../org/apache/hadoop/yarn/sls/TestSLSRunner.java | 39 +
.../hadoop-yarn/hadoop-yarn-api/pom.xml | 41 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 22 +
.../src/main/proto/YarnCsiAdaptor.proto | 1 +
.../main/proto/application_history_client.proto | 1 +
.../main/proto/applicationclient_protocol.proto | 1 +
.../main/proto/applicationmaster_protocol.proto | 1 +
.../src/main/proto/client_SCM_protocol.proto | 1 +
.../main/proto/containermanagement_protocol.proto | 1 +
.../src/main/proto/server/SCM_Admin_protocol.proto | 1 +
.../proto/server/application_history_server.proto | 1 +
.../resourcemanager_administration_protocol.proto | 1 +
...arn_server_resourcemanager_service_protos.proto | 1 +
.../src/main/proto/yarn_csi_adaptor.proto | 4 +-
.../src/main/proto/yarn_protos.proto | 1 +
.../src/main/proto/yarn_service_protos.proto | 1 +
.../hadoop-yarn-services-core/pom.xml | 23 +-
.../src/main/proto/ClientAMProtocol.proto | 1 +
.../hadoop-yarn/hadoop-yarn-client/pom.xml | 41 +-
.../hadoop/yarn/client/cli/SchedConfCLI.java | 147 +++-
.../apache/hadoop/yarn/client/cli/TestLogsCLI.java | 59 +-
.../hadoop/yarn/client/cli/TestSchedConfCLI.java | 187 ++++-
.../src/test/proto/test_amrm_token.proto | 1 +
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 41 +-
.../hadoop/yarn/logaggregation/LogToolUtils.java | 60 +-
.../src/main/proto/yarn_security_token.proto | 1 +
.../src/main/resources/yarn-default.xml | 9 +
.../logaggregation/TestAggregatedLogFormat.java | 42 +-
.../TestLogAggregationIndexedFileController.java | 31 +-
.../pom.xml | 39 +-
.../yarn_server_timelineserver_recovery.proto | 1 +
.../hadoop-yarn-server-common/pom.xml | 39 +-
...DistributedOpportunisticContainerAllocator.java | 357 +++++++++
.../scheduler/OpportunisticContainerAllocator.java | 347 ++-------
.../src/main/proto/ResourceTracker.proto | 1 +
.../src/main/proto/SCMUploader.proto | 1 +
.../main/proto/collectornodemanager_protocol.proto | 1 +
.../proto/distributed_scheduling_am_protocol.proto | 2 +-
.../src/main/proto/yarn_server_common_protos.proto | 1 +
.../proto/yarn_server_common_service_protos.proto | 1 +
.../main/proto/yarn_server_federation_protos.proto | 1 +
.../TestOpportunisticContainerAllocator.java | 2 +-
.../hadoop-yarn-server-nodemanager/pom.xml | 34 +-
.../yarn/server/nodemanager/NodeManager.java | 3 +-
.../src/main/proto/LocalizationProtocol.proto | 1 +
.../proto/yarn_server_nodemanager_recovery.proto | 1 +
.../yarn_server_nodemanager_service_protos.proto | 1 +
.../logaggregation/TestLogAggregationService.java | 106 +--
.../scheduler/TestDistributedScheduler.java | 4 +-
.../nodemanager/webapp/TestNMWebServices.java | 43 +-
.../hadoop-yarn-server-resourcemanager/pom.xml | 69 +-
.../server/resourcemanager/ClientRMService.java | 6 +-
.../resourcemanager/DefaultAMSProcessor.java | 8 +
.../OpportunisticContainerAllocatorAMService.java | 30 +-
.../yarn/server/resourcemanager/RMAppManager.java | 7 +
.../yarn/server/resourcemanager/RMAuditLogger.java | 53 +-
.../scheduler/MutableConfigurationProvider.java | 2 +
.../scheduler/SchedulerApplicationAttempt.java | 9 +
.../resourcemanager/scheduler/SchedulerUtils.java | 23 +
.../capacity/CapacitySchedulerConfiguration.java | 7 +
.../scheduler/capacity/LeafQueue.java | 8 +-
.../conf/FSSchedulerConfigurationStore.java | 5 +
.../capacity/conf/InMemoryConfigurationStore.java | 5 +
.../capacity/conf/LeveldbConfigurationStore.java | 7 +
.../conf/MutableCSConfigurationProvider.java | 46 +-
.../capacity/conf/YarnConfigurationStore.java | 7 +
.../capacity/conf/ZKConfigurationStore.java | 5 +
.../policy/AbstractComparatorOrderingPolicy.java | 8 +-
.../FifoOrderingPolicyWithExclusivePartitions.java | 144 ++++
.../scheduler/policy/IteratorSelector.java} | 37 +-
.../scheduler/policy/OrderingPolicy.java | 3 +-
.../scheduler/policy/SchedulableEntity.java | 5 +
.../server/resourcemanager/webapp/RMWSConsts.java | 3 +
.../resourcemanager/webapp/RMWebServices.java | 31 +
.../yarn_server_resourcemanager_recovery.proto | 1 +
.../server/resourcemanager/TestRMAuditLogger.java | 21 +-
.../scheduler/TestSchedulerUtils.java | 142 ++++
.../scheduler/capacity/TestCapacityScheduler.java | 11 +-
.../scheduler/capacity/TestLeafQueue.java | 145 +++-
.../conf/TestFSSchedulerConfigurationStore.java | 16 +
.../conf/TestMutableCSConfigurationProvider.java | 8 +
.../capacity/conf/TestZKConfigurationStore.java | 9 +
.../scheduler/policy/MockSchedulableEntity.java | 15 +-
.../scheduler/policy/TestFairOrderingPolicy.java | 12 +-
.../scheduler/policy/TestFifoOrderingPolicy.java | 2 +-
.../TestFifoOrderingPolicyForPendingApps.java | 5 +-
...tFifoOrderingPolicyWithExclusivePartitions.java | 244 ++++++
.../security/TestDelegationTokenRenewer.java | 4 +
.../TestRMWebServicesConfigurationMutation.java | 13 +
.../src/test/proto/test_client_tokens.proto | 1 +
.../hadoop-yarn-server-tests/pom.xml | 31 +-
.../src/test/proto/test_token.proto | 1 +
.../src/site/markdown/NodeManager.md | 8 +-
.../src/site/markdown/ResourceManagerRest.md | 81 ++
pom.ozone.xml | 29 +-
520 files changed, 18016 insertions(+), 4042 deletions(-)
copy hadoop-hdfs-project/hadoop-hdfs-nfs/dev-support/findbugsExcludeFile.xml => hadoop-cloud-storage-project/hadoop-cos/dev-support/findbugs-exclude.xml (100%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/pom.xml
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/site/markdown/cloud-storage/index.md
copy {hadoop-tools/hadoop-resourceestimator/src => hadoop-cloud-storage-project/hadoop-cos}/site/resources/css/site.css (100%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/BufferPool.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferInputStream.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferOutputStream.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/ByteBufferWrapper.java
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Constants.java (50%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosN.java (56%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNConfigKeys.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileContext.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNCopyFileTask.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileReadTask.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNInputStream.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNOutputStream.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNUtils.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/FileMetadata.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/NativeFileSystemStore.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/PartialListing.java
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/Unit.java (64%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/COSCredentialProviderList.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/EnvironmentVariableCredentialProvider.java
copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/proto/test_amrm_token.proto => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/NoAuthWithCOSException.java (59%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/SimpleCredentialProvider.java
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/package-info.java (66%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/package-info.java (66%)
copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/proto/test_amrm_token.proto => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestConfigKey.java (60%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/CosNTestUtils.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNInputStream.java
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/TestCosNOutputStream.java
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/CosNContract.java (51%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractCreate.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDelete.java (59%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractDistCp.java
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractGetFileStatus.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractMkdir.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractOpen.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRename.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractRootDir.java (59%)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/TestCosNContractSeek.java (59%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-cloud-storage-project/hadoop-cos/src/test/java/org/apache/hadoop/fs/cosn/contract/package-info.java (66%)
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/resources/contract/cosn.xml
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/resources/core-site.xml
create mode 100644 hadoop-cloud-storage-project/hadoop-cos/src/test/resources/log4j.properties
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java
create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FsLinkResolution.java
copy hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementStatus.java => hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/PathCapabilitiesSupport.java (50%)
create mode 100644 hadoop-common-project/hadoop-common/src/site/markdown/filesystem/pathcapabilities.md
create mode 100644 hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/CHANGES.3.1.3.md
create mode 100644 hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.3/RELEASENOTES.3.1.3.md
copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/SCM_Admin_protocol.proto => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/FunctionWithServiceException.java (60%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/package-info.java (71%)
delete mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBConfiguration.java
delete mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerLocationProtocolServerSideTranslatorPB.java
create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/package-info.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/SCMSecurityProtocolServerSideTranslatorPB.java
rename hadoop-hdds/{common/src/main/java/org/apache/hadoop/ozone/protocolPB => server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol}/ScmBlockLocationProtocolServerSideTranslatorPB.java (85%)
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/package-info.java (66%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/{CloseSubcommand.java => ContainerCommands.java} (66%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java => pipeline/PipelineCommands.java} (63%)
create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/DfsPathCapabilities.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff/Apache_Hadoop_HDFS_3.1.3.xml
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementStatusDefault.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/RepeatedOmKeyInfoCodec.java
rename hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/{VolumeListCodec.java => UserVolumeInfoCodec.java} (78%)
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java
copy hadoop-ozone/dist/src/main/compose/{test-single.sh => ozonesecure-mr/test.sh} (56%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/gdpr/gdpr.robot
copy hadoop-ozone/dist/src/main/smoketest/{kinit.robot => kinit-hadoop.robot} (94%)
copy hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java => ScmProtocolContainerLocationInsight.java} (73%)
copy hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java => ScmProtocolDatanodeInsight.java} (67%)
copy hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java => ScmProtocolSecurityInsight.java} (75%)
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerRocksDBLogging.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ha/OMHANodeDetails.java
rename hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/{ => ha}/OMNodeDetails.java (99%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ha/package-info.java (71%)
copy hadoop-ozone/{s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/OzoneS3Util.java => ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/package-info.java} (68%)
copy hadoop-ozone/{s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/OzoneS3Util.java => ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/package-info.java} (68%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/volume/package-info.java (66%)
create mode 100644 hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestOzoneS3Util.java
copy hadoop-ozone/{ozonefs/src/test => tools/src/main}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem (100%)
create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AwsSignerInitializer.java
rename hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/{ => auth}/SignerManager.java (50%)
copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/proto/ShuffleHandlerRecovery.proto => hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationTokenProvider.java (69%)
copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/proto/test_amrm_token.proto => hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestLocatedFileStatusFetcher.java (59%)
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFSMainOperations.java
delete mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestSignerManager.java
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/TestSignerManager.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/DistributedOpportunisticContainerAllocator.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicyWithExclusivePartitions.java
copy hadoop-yarn-project/hadoop-yarn/{hadoop-yarn-client/src/test/proto/test_amrm_token.proto => hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/IteratorSelector.java} (55%)
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFifoOrderingPolicyWithExclusivePartitions.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/01: Merge remote-tracking branch 'origin/trunk' into
HDDS-1880-Decom
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit ec70207838d5b29fa0b534b13c103865e50a35e8
Merge: fd5e877 6171a41
Author: Márton Elek <el...@apache.org>
AuthorDate: Fri Oct 4 14:17:38 2019 +0200
Merge remote-tracking branch 'origin/trunk' into HDDS-1880-Decom
BUILDING.txt | 31 -
dev-support/docker/Dockerfile | 19 +-
.../hadoop-cos/dev-support/findbugs-exclude.xml | 18 +
hadoop-cloud-storage-project/hadoop-cos/pom.xml | 140 ++++
.../site/markdown/cloud-storage/index.md | 367 ++++++++++
.../hadoop-cos/site/resources/css/site.css | 29 +
.../java/org/apache/hadoop/fs/cosn/BufferPool.java | 245 +++++++
.../hadoop/fs/cosn/ByteBufferInputStream.java | 89 +++
.../hadoop/fs/cosn/ByteBufferOutputStream.java | 74 ++
.../apache/hadoop/fs/cosn/ByteBufferWrapper.java | 103 +++
.../java/org/apache/hadoop/fs/cosn/Constants.java | 35 +-
.../main/java/org/apache/hadoop/fs/cosn/CosN.java | 31 +-
.../org/apache/hadoop/fs/cosn/CosNConfigKeys.java | 86 +++
.../apache/hadoop/fs/cosn/CosNCopyFileContext.java | 66 ++
.../apache/hadoop/fs/cosn/CosNCopyFileTask.java | 68 ++
.../apache/hadoop/fs/cosn/CosNFileReadTask.java | 125 ++++
.../org/apache/hadoop/fs/cosn/CosNFileSystem.java | 814 +++++++++++++++++++++
.../org/apache/hadoop/fs/cosn/CosNInputStream.java | 365 +++++++++
.../apache/hadoop/fs/cosn/CosNOutputStream.java | 284 +++++++
.../java/org/apache/hadoop/fs/cosn/CosNUtils.java | 167 +++++
.../hadoop/fs/cosn/CosNativeFileSystemStore.java | 768 +++++++++++++++++++
.../org/apache/hadoop/fs/cosn/FileMetadata.java | 68 ++
.../hadoop/fs/cosn/NativeFileSystemStore.java | 99 +++
.../org/apache/hadoop/fs/cosn/PartialListing.java | 64 ++
.../main/java/org/apache/hadoop/fs/cosn/Unit.java | 27 +-
.../fs/cosn/auth/COSCredentialProviderList.java | 139 ++++
.../EnvironmentVariableCredentialProvider.java | 55 ++
.../fs/cosn/auth/NoAuthWithCOSException.java | 32 +-
.../fs/cosn/auth/SimpleCredentialProvider.java | 54 ++
.../apache/hadoop/fs/cosn/auth/package-info.java | 19 +-
.../org/apache/hadoop/fs/cosn/package-info.java | 19 +-
.../apache/hadoop/fs/cosn/CosNTestConfigKey.java | 30 +-
.../org/apache/hadoop/fs/cosn/CosNTestUtils.java | 78 ++
.../apache/hadoop/fs/cosn/TestCosNInputStream.java | 167 +++++
.../hadoop/fs/cosn/TestCosNOutputStream.java | 87 +++
.../hadoop/fs/cosn/contract/CosNContract.java | 36 +-
.../fs/cosn/contract/TestCosNContractCreate.java | 26 +-
.../fs/cosn/contract/TestCosNContractDelete.java | 26 +-
.../fs/cosn/contract/TestCosNContractDistCp.java | 54 ++
.../contract/TestCosNContractGetFileStatus.java | 27 +-
.../fs/cosn/contract/TestCosNContractMkdir.java | 26 +-
.../fs/cosn/contract/TestCosNContractOpen.java | 26 +-
.../fs/cosn/contract/TestCosNContractRename.java | 26 +-
.../fs/cosn/contract/TestCosNContractRootDir.java | 27 +-
.../fs/cosn/contract/TestCosNContractSeek.java | 26 +-
.../hadoop/fs/cosn/contract/package-info.java | 19 +-
.../src/test/resources/contract/cosn.xml | 120 +++
.../hadoop-cos/src/test/resources/core-site.xml | 107 +++
.../hadoop-cos/src/test/resources/log4j.properties | 18 +
hadoop-cloud-storage-project/pom.xml | 1 +
.../apache/hadoop/crypto/CryptoInputStream.java | 67 +-
.../org/apache/hadoop/fs/AbstractFileSystem.java | 16 +-
.../hadoop/fs/ByteBufferPositionedReadable.java | 24 +
.../org/apache/hadoop/fs/ChecksumFileSystem.java | 22 +
.../apache/hadoop/fs/CommonPathCapabilities.java | 126 ++++
.../org/apache/hadoop/fs/DelegateToFileSystem.java | 7 +
.../org/apache/hadoop/fs/FSDataInputStream.java | 23 +-
.../java/org/apache/hadoop/fs/FileContext.java | 23 +-
.../main/java/org/apache/hadoop/fs/FileSystem.java | 30 +-
.../org/apache/hadoop/fs/FilterFileSystem.java | 7 +
.../main/java/org/apache/hadoop/fs/FilterFs.java | 5 +
.../main/java/org/apache/hadoop/fs/Globber.java | 208 +++++-
.../java/org/apache/hadoop/fs/HarFileSystem.java | 19 +-
.../org/apache/hadoop/fs/PathCapabilities.java | 61 ++
.../org/apache/hadoop/fs/RawLocalFileSystem.java | 19 +
.../hadoop/fs/http/AbstractHttpFileSystem.java | 18 +
.../apache/hadoop/fs/impl/FsLinkResolution.java | 98 +++
.../hadoop/fs/impl/PathCapabilitiesSupport.java | 40 +-
.../java/org/apache/hadoop/fs/shell/Mkdir.java | 4 +-
.../hadoop/fs/viewfs/ChRootedFileSystem.java | 6 +
.../apache/hadoop/fs/viewfs/ViewFileSystem.java | 32 +
.../apache/hadoop/util/NodeHealthScriptRunner.java | 1 +
.../src/site/markdown/DeprecatedProperties.md | 4 +
.../src/site/markdown/filesystem/filesystem.md | 5 +-
.../src/site/markdown/filesystem/index.md | 1 +
.../site/markdown/filesystem/pathcapabilities.md | 158 ++++
.../site/markdown/release/3.1.3/CHANGES.3.1.3.md | 336 +++++++++
.../markdown/release/3.1.3/RELEASENOTES.3.1.3.md | 59 ++
.../hadoop/crypto/CryptoStreamsTestBase.java | 115 ++-
.../apache/hadoop/crypto/TestCryptoStreams.java | 24 +
.../hadoop/crypto/TestCryptoStreamsForLocalFS.java | 5 +
.../hadoop/crypto/TestCryptoStreamsNormal.java | 5 +
.../fs/contract/AbstractContractAppendTest.java | 10 +
.../fs/contract/AbstractContractConcatTest.java | 7 +
.../hadoop/fs/contract/ContractTestUtils.java | 50 +-
.../org/apache/hadoop/test/LambdaTestUtils.java | 3 +
.../hadoop/util/TestNodeHealthScriptRunner.java | 9 +
hadoop-hdds/common/pom.xml | 4 +-
.../function/FunctionWithServiceException.java | 29 +-
.../apache/hadoop/hdds/function/package-info.java | 15 +-
.../SCMSecurityProtocolClientSideTranslatorPB.java | 104 ++-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 132 ----
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
...inerLocationProtocolClientSideTranslatorPB.java | 411 +++++------
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 24 +-
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 62 ++
.../apache/hadoop/hdds/utils/db/TypedTable.java | 2 +-
.../hadoop/hdds/utils/db/cache/TableCache.java | 9 +
.../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 7 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../org/apache/hadoop/ozone/lock/ActiveLock.java | 63 +-
.../org/apache/hadoop/ozone/lock/LockManager.java | 166 ++++-
...inerLocationProtocolServerSideTranslatorPB.java | 389 ----------
.../src/main/proto/SCMSecurityProtocol.proto | 96 ++-
.../src/main/proto/ScmBlockLocationProtocol.proto | 2 +-
.../proto/StorageContainerLocationProtocol.proto | 185 +++--
.../common/src/main/resources/ozone-default.xml | 8 +
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 16 +-
.../apache/hadoop/ozone/lock/TestLockManager.java | 145 +++-
.../hadoop/hdds/conf/ConfigFileGenerator.java | 5 +-
hadoop-hdds/container-service/pom.xml | 9 +-
.../server/ratis/ContainerStateMachine.java | 20 +-
.../transport/server/ratis/XceiverServerRatis.java | 8 +-
.../replication/GrpcReplicationClient.java | 6 +
...inerDatanodeProtocolClientSideTranslatorPB.java | 60 +-
...inerDatanodeProtocolServerSideTranslatorPB.java | 115 +--
.../proto/StorageContainerDatanodeProtocol.proto | 58 +-
.../ozone/container/common/SCMTestUtils.java | 4 +-
.../server/OzoneProtocolMessageDispatcher.java | 88 +++
.../apache/hadoop/hdds/server/ProfileServlet.java | 10 +-
.../hadoop/hdds/server/PrometheusMetricsSink.java | 16 +-
.../hdds/server/TestPrometheusMetricsSink.java | 77 +-
hadoop-hdds/pom.xml | 17 +-
hadoop-hdds/server-scm/pom.xml | 4 +-
.../container/AbstractContainerReportHandler.java | 8 +
.../hdds/scm/container/ContainerManager.java | 8 +
.../hdds/scm/container/ContainerReportHandler.java | 2 +
.../IncrementalContainerReportHandler.java | 10 +
.../hdds/scm/container/SCMContainerManager.java | 48 +-
.../metrics/SCMContainerManagerMetrics.java | 144 ++++
.../hdds/scm/container/metrics/package-info.java | 22 +
.../apache/hadoop/hdds/scm/node/NodeManager.java | 8 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 51 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 186 +++++
...lockLocationProtocolServerSideTranslatorPB.java | 56 +-
...inerLocationProtocolServerSideTranslatorPB.java | 393 ++++++++++
.../hadoop/hdds/scm/protocol/package-info.java | 22 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 26 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 9 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 15 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 102 ++-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 27 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 36 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 67 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 6 +-
.../scm/server/TestSCMBlockProtocolServer.java | 3 +-
.../testutils/ReplicationNodeManagerMock.java | 5 +-
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 22 +-
.../hdds/scm/cli/container/CloseSubcommand.java | 7 +-
...CloseSubcommand.java => ContainerCommands.java} | 37 +-
.../hdds/scm/cli/container/CreateSubcommand.java | 5 +-
.../hdds/scm/cli/container/DeleteSubcommand.java | 7 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 5 +-
.../hdds/scm/cli/container/ListSubcommand.java | 5 +-
.../cli/pipeline/ActivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ClosePipelineSubcommand.java | 11 +-
.../cli/pipeline/DeactivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ListPipelinesSubcommand.java | 11 +-
.../PipelineCommands.java} | 38 +-
.../org/apache/hadoop/hdfs/DFSInputStream.java | 14 +
.../apache/hadoop/hdfs/DistributedFileSystem.java | 22 +
.../hadoop/hdfs/client/DfsPathCapabilities.java | 62 ++
.../org/apache/hadoop/hdfs/protocol/Block.java | 141 +++-
.../namenode/ha/ObserverReadProxyProvider.java | 26 +-
.../apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 30 +
.../hadoop/hdfs/TestDefaultNameNodePort.java | 38 +-
.../hadoop/fs/http/client/HttpFSFileSystem.java | 30 +
.../fs/http/server/HttpFSAuthenticationFilter.java | 16 +-
.../fs/http/server/HttpFSServerWebServer.java | 22 +
.../service/hadoop/FileSystemAccessService.java | 21 +-
.../src/main/resources/httpfs-default.xml | 20 +-
.../main/native/libhdfs-tests/test_libhdfs_ops.c | 43 +-
.../src/main/native/libhdfs/hdfs.c | 117 +++
.../src/main/native/libhdfs/include/hdfs/hdfs.h | 21 +-
.../src/main/native/libhdfspp/tests/hdfs_shim.c | 6 +
.../libhdfspp/tests/libhdfs_wrapper_defines.h | 1 +
.../libhdfspp/tests/libhdfs_wrapper_undefs.h | 1 +
.../libhdfspp/tests/libhdfspp_wrapper_defines.h | 1 +
hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 48 ++
.../src/main/webapps/router/federationhealth.html | 10 +-
.../src/main/webapps/router/federationhealth.js | 7 +-
.../src/main/webapps/static/rbf.css | 26 +-
.../fs/contract/router/RouterHDFSContract.java | 17 +-
.../fs/contract/router/SecurityConfUtil.java | 15 +-
.../router/TestRouterHDFSContractAppendSecure.java | 4 +-
.../router/TestRouterHDFSContractConcatSecure.java | 5 +-
.../router/TestRouterHDFSContractCreateSecure.java | 5 +-
.../TestRouterHDFSContractDelegationToken.java | 2 +-
.../router/TestRouterHDFSContractDeleteSecure.java | 4 +-
.../TestRouterHDFSContractGetFileStatusSecure.java | 5 +-
.../router/TestRouterHDFSContractMkdirSecure.java | 5 +-
.../router/TestRouterHDFSContractOpenSecure.java | 5 +-
.../router/TestRouterHDFSContractRenameSecure.java | 5 +-
.../TestRouterHDFSContractRootDirectorySecure.java | 5 +-
.../router/TestRouterHDFSContractSeekSecure.java | 5 +-
.../TestRouterHDFSContractSetTimesSecure.java | 5 +-
.../server/federation/router/TestRouterQuota.java | 2 +-
.../security/TestRouterHttpDelegationToken.java | 1 +
.../dev-support/jdiff/Apache_Hadoop_HDFS_3.1.3.xml | 676 +++++++++++++++++
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 11 +-
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +-
.../hadoop/hdfs/qjournal/client/QuorumCall.java | 18 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 2 +
.../token/block/BlockTokenSecretManager.java | 31 +-
.../hdfs/server/blockmanagement/BlockInfo.java | 6 +
.../blockmanagement/BlockInfoContiguous.java | 14 +
.../server/blockmanagement/BlockInfoStriped.java | 9 +
.../hdfs/server/blockmanagement/BlockManager.java | 94 ++-
.../blockmanagement/BlockPlacementStatus.java | 8 +
.../BlockPlacementStatusDefault.java | 8 +
.../BlockPlacementStatusWithNodeGroup.java | 11 +
.../BlockPlacementStatusWithUpgradeDomain.java | 22 +-
.../blockmanagement/BlockReportLeaseManager.java | 8 +-
.../hdfs/server/namenode/INodeDirectory.java | 6 +
.../hadoop/hdfs/server/namenode/INodeFile.java | 3 +
.../hdfs/server/namenode/NameNodeRpcServer.java | 14 +-
.../namenode/snapshot/AbstractINodeDiffList.java | 4 +
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 2 +-
.../src/main/resources/hdfs-default.xml | 19 +
.../src/main/webapps/hdfs/dfshealth.html | 4 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 4 +-
.../src/site/markdown/ObserverNameNode.md | 14 +
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 17 +-
.../hadoop/hdfs/TestBlockMissingException.java | 2 -
.../apache/hadoop/hdfs/TestByteBufferPread.java | 27 +-
.../java/org/apache/hadoop/hdfs/TestClose.java | 1 -
.../hdfs/TestStateAlignmentContextWithHA.java | 1 +
.../hadoop/hdfs/qjournal/MiniJournalCluster.java | 6 +-
.../qjournal/client/TestQuorumJournalManager.java | 34 +-
.../hdfs/security/token/block/TestBlockToken.java | 23 +
.../blockmanagement/BlockManagerTestUtil.java | 30 +-
.../hdfs/server/blockmanagement/TestBlockInfo.java | 36 +
.../server/blockmanagement/TestBlockManager.java | 58 ++
.../TestBlockPlacementStatusDefault.java | 57 ++
.../TestBlockPlacementStatusWithUpgradeDomain.java | 59 +-
.../TestBlocksWithNotEnoughRacks.java | 111 ++-
.../namenode/ha/TestConsistentReadsObserver.java | 2 +
.../ha/TestFailoverWithBlockTokensEnabled.java | 36 +-
.../server/namenode/ha/TestMultiObserverNode.java | 2 +
.../hdfs/server/namenode/ha/TestObserverNode.java | 2 +
.../namenode/snapshot/TestRenameWithSnapshots.java | 15 +-
.../namenode/snapshot/TestSnapshotDeletion.java | 2 +-
.../hadoop-mapreduce-client-common/pom.xml | 34 +-
.../src/main/proto/HSAdminRefreshProtocol.proto | 3 +-
.../src/main/proto/MRClientProtocol.proto | 1 +
.../src/main/proto/mr_protos.proto | 1 +
.../src/main/proto/mr_service_protos.proto | 1 +
.../org/apache/hadoop/mapred/FileInputFormat.java | 5 +-
.../hadoop/mapred/InvalidInputException.java | 4 +
.../hadoop/mapred/LocatedFileStatusFetcher.java | 66 +-
.../mapreduce/lib/input/FileInputFormat.java | 6 +-
.../mapreduce/lib/input/InvalidInputException.java | 4 +
.../hadoop-mapreduce-client-shuffle/pom.xml | 27 +-
.../src/main/proto/ShuffleHandlerRecovery.proto | 1 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 3 +-
hadoop-ozone/common/pom.xml | 4 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 57 +-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 8 +-
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 52 ++
...lumeListCodec.java => UserVolumeInfoCodec.java} | 13 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 100 ++-
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 91 +++
.../hadoop/ozone/security/GDPRSymmetricKey.java | 8 +-
.../OzoneDelegationTokenSecretManager.java | 5 +-
.../hadoop/ozone/security/OzoneSecretManager.java | 7 +-
.../src/main/proto/OzoneManagerProtocol.proto | 18 +-
.../ozone/security/TestGDPRSymmetricKey.java | 4 +-
hadoop-ozone/csi/pom.xml | 4 +-
.../dev-support/checks/_mvn_unit_report.sh | 16 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 2 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 12 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 39 +-
.../src/main/compose/ozonesecure-mr/docker-config | 31 +-
.../{test-single.sh => ozonesecure-mr/test.sh} | 37 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 4 +-
hadoop-ozone/dist/src/main/compose/test-single.sh | 2 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 27 +-
.../dist/src/main/smoketest/gdpr/gdpr.robot | 89 +++
.../smoketest/{kinit.robot => kinit-hadoop.robot} | 2 +-
hadoop-ozone/dist/src/main/smoketest/kinit.robot | 5 +-
.../dist/src/main/smoketest/mapreduce.robot | 2 +-
hadoop-ozone/insight/pom.xml | 9 +-
.../ozone/insight/BaseInsightSubCommand.java | 7 +-
.../scm/ScmProtocolBlockLocationInsight.java | 6 +-
...va => ScmProtocolContainerLocationInsight.java} | 18 +-
...nsight.java => ScmProtocolDatanodeInsight.java} | 27 +-
...nsight.java => ScmProtocolSecurityInsight.java} | 18 +-
.../metrics/TestSCMContainerManagerMetrics.java | 167 +++++
.../scm/pipeline/TestRatisPipelineProvider.java | 26 +-
.../rpc/TestContainerStateMachineFailures.java | 65 ++
.../client/rpc/TestOzoneRpcClientAbstract.java | 76 +-
.../ozone/om/TestOzoneManagerConfiguration.java | 9 +-
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 97 +++
hadoop-ozone/ozone-manager/pom.xml | 5 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 40 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 113 +--
.../org/apache/hadoop/ozone/om/OzoneManager.java | 232 +-----
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 35 +-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 306 ++++++++
.../hadoop/ozone/om/{ => ha}/OMNodeDetails.java | 2 +-
.../apache/hadoop/ozone/om/ha/package-info.java | 14 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 3 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 8 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 4 +-
.../om/request/volume/OMVolumeCreateRequest.java | 12 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 5 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 32 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 10 +-
.../om/response/bucket/OMBucketCreateResponse.java | 8 +-
.../om/response/bucket/OMBucketDeleteResponse.java | 4 +-
.../bucket/OMBucketSetPropertyResponse.java | 7 +-
.../response/file/OMDirectoryCreateResponse.java | 3 +-
.../om/response/file/OMFileCreateResponse.java | 3 +-
.../om/response/key/OMAllocateBlockResponse.java | 6 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 7 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 3 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 32 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 4 +-
.../ozone/om/response/key/OMKeyRenameResponse.java | 6 +-
.../multipart/S3MultipartUploadAbortResponse.java | 21 +-
.../S3MultipartUploadCommitPartResponse.java | 45 +-
.../S3MultipartUploadCompleteResponse.java | 3 +-
.../om/response/volume/OMVolumeAclOpResponse.java | 3 +-
.../om/response/volume/OMVolumeCreateResponse.java | 13 +-
.../om/response/volume/OMVolumeDeleteResponse.java | 10 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 12 +-
.../response/volume/OMVolumeSetQuotaResponse.java | 4 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 2 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 41 +-
.../org/apache/hadoop/ozone/web/ozShell/Shell.java | 3 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 21 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/TestOMRequestUtils.java | 25 +-
.../hadoop/ozone/om/request/package-info.java} | 18 +-
.../request/volume/TestOMVolumeCreateRequest.java | 16 +-
.../volume/TestOMVolumeSetOwnerRequest.java | 4 +-
.../ozone/om/response/TestOMResponseUtils.java | 8 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 20 +-
.../hadoop/ozone/om/response/package-info.java} | 18 +-
.../s3/multipart/TestS3MultipartResponse.java | 3 +-
.../TestS3MultipartUploadAbortResponse.java | 19 +-
.../volume/TestOMVolumeCreateResponse.java | 7 +-
.../volume/TestOMVolumeDeleteResponse.java | 11 +-
.../volume/TestOMVolumeSetOwnerResponse.java | 17 +-
.../ozone/om/response/volume/package-info.java | 22 +-
.../TestOzoneDelegationTokenSecretManager.java | 29 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 4 +-
hadoop-ozone/ozonefs-lib-legacy/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 9 +-
hadoop-ozone/pom.xml | 14 +-
hadoop-ozone/recon/pom.xml | 4 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 35 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 21 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 28 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 33 +-
.../recon/spi/impl/ReconContainerDBProvider.java | 32 +-
.../ozone/recon/AbstractOMMetadataManagerTest.java | 2 +-
.../apache/hadoop/ozone/recon/TestReconUtils.java | 31 +
.../recovery/TestReconOmMetadataManagerImpl.java | 133 ++--
.../impl/TestOzoneManagerServiceProviderImpl.java | 29 +-
.../spi/impl/TestReconContainerDBProvider.java | 13 -
hadoop-ozone/s3gateway/pom.xml | 5 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 11 +-
.../hadoop/ozone/s3/OzoneServiceProvider.java | 50 +-
.../apache/hadoop/ozone/s3/util/OzoneS3Util.java | 44 ++
.../hadoop/ozone/s3/util/TestOzoneS3Util.java | 130 ++++
hadoop-ozone/tools/pom.xml | 9 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 3 +-
.../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 4 +-
.../services/org.apache.hadoop.fs.FileSystem} | 11 +-
hadoop-ozone/upgrade/pom.xml | 5 +-
hadoop-project/pom.xml | 4 +-
hadoop-project/src/site/site.xml | 1 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 28 +-
.../java/org/apache/hadoop/fs/s3a/Invoker.java | 3 +-
.../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 6 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 76 +-
.../hadoop/fs/s3a/auth/AwsSignerInitializer.java | 53 ++
.../hadoop/fs/s3a/{ => auth}/SignerManager.java | 80 +-
.../auth/delegation/DelegationTokenProvider.java | 20 +-
.../hadoop/fs/s3a/commit/CommitConstants.java | 22 +-
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 3 +-
.../hadoop/fs/s3a/select/SelectConstants.java | 2 +-
.../apache/hadoop/fs/s3a/select/SelectTool.java | 2 +-
.../src/site/markdown/tools/hadoop-aws/index.md | 58 ++
.../fs/s3a/ITestLocatedFileStatusFetcher.java | 31 +-
.../hadoop/fs/s3a/ITestS3AFSMainOperations.java | 65 ++
.../hadoop/fs/s3a/ITestS3AMiscOperations.java | 7 +
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 7 +-
.../apache/hadoop/fs/s3a/TestSignerManager.java | 130 ----
.../hadoop/fs/s3a/auth/ITestCustomSigner.java | 237 ++++++
.../fs/s3a/auth/ITestRestrictedReadAccess.java | 707 ++++++++++++++++++
.../hadoop/fs/s3a/auth/TestSignerManager.java | 590 +++++++++++++++
.../fs/s3a/commit/ITestCommitOperations.java | 5 +-
.../s3a/s3guard/AbstractS3GuardToolTestBase.java | 2 +-
.../apache/hadoop/fs/s3a/select/ITestS3Select.java | 4 +-
.../org/apache/hadoop/fs/adl/AdlFileSystem.java | 18 +
.../fs/adl/live/TestAdlSdkConfiguration.java | 1 +
.../hadoop/fs/azure/NativeAzureFileSystem.java | 17 +
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 14 +
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 22 +-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 6 +-
.../fs/azurebfs/constants/ConfigurationKeys.java | 8 +
.../constants/FileSystemConfigurations.java | 1 +
.../fs/azurebfs/services/AbfsOutputStream.java | 7 +-
.../hadoop-azure/src/site/markdown/abfs.md | 17 +
.../fs/azurebfs/ITestAzureBlobFileSystemFlush.java | 27 +-
.../fs/azurebfs/ITestGetNameSpaceEnabled.java | 6 +-
.../org/apache/hadoop/tools/DistCpConstants.java | 6 +
.../apache/hadoop/tools/mapred/CopyCommitter.java | 10 +-
.../org/apache/hadoop/tools/mapred/CopyMapper.java | 3 +-
.../tools/mapred/RetriableFileCopyCommand.java | 34 +-
.../org/apache/hadoop/tools/util/DistCpUtils.java | 26 +-
.../hadoop/tools/mapred/TestCopyCommitter.java | 5 +-
.../apache/hadoop/tools/mapred/TestCopyMapper.java | 56 ++
.../tools/mapred/TestRetriableFileCopyCommand.java | 25 +-
.../apache/hadoop/tools/util/TestDistCpUtils.java | 45 +-
.../tools/util/TestDistCpUtilsWithCombineMode.java | 4 +-
.../BlockPlacementPolicyAlwaysSatisfied.java | 5 +
.../java/org/apache/hadoop/yarn/sls/SLSRunner.java | 25 +
.../hadoop/yarn/sls/conf/SLSConfiguration.java | 3 +
.../apache/hadoop/yarn/sls/BaseSLSRunnerTest.java | 4 +-
.../org/apache/hadoop/yarn/sls/TestSLSRunner.java | 39 +
.../hadoop-yarn/hadoop-yarn-api/pom.xml | 41 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 22 +
.../src/main/proto/YarnCsiAdaptor.proto | 1 +
.../main/proto/application_history_client.proto | 1 +
.../main/proto/applicationclient_protocol.proto | 1 +
.../main/proto/applicationmaster_protocol.proto | 1 +
.../src/main/proto/client_SCM_protocol.proto | 1 +
.../main/proto/containermanagement_protocol.proto | 1 +
.../src/main/proto/server/SCM_Admin_protocol.proto | 1 +
.../proto/server/application_history_server.proto | 1 +
.../resourcemanager_administration_protocol.proto | 1 +
...arn_server_resourcemanager_service_protos.proto | 1 +
.../src/main/proto/yarn_csi_adaptor.proto | 4 +-
.../src/main/proto/yarn_protos.proto | 1 +
.../src/main/proto/yarn_service_protos.proto | 1 +
.../hadoop-yarn-services-core/pom.xml | 23 +-
.../src/main/proto/ClientAMProtocol.proto | 1 +
.../hadoop-yarn/hadoop-yarn-client/pom.xml | 41 +-
.../hadoop/yarn/client/cli/SchedConfCLI.java | 147 +++-
.../apache/hadoop/yarn/client/cli/TestLogsCLI.java | 59 +-
.../hadoop/yarn/client/cli/TestSchedConfCLI.java | 187 ++++-
.../src/test/proto/test_amrm_token.proto | 1 +
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 41 +-
.../hadoop/yarn/logaggregation/LogToolUtils.java | 60 +-
.../src/main/proto/yarn_security_token.proto | 1 +
.../src/main/resources/yarn-default.xml | 9 +
.../logaggregation/TestAggregatedLogFormat.java | 42 +-
.../TestLogAggregationIndexedFileController.java | 31 +-
.../pom.xml | 39 +-
.../yarn_server_timelineserver_recovery.proto | 1 +
.../hadoop-yarn-server-common/pom.xml | 39 +-
...DistributedOpportunisticContainerAllocator.java | 357 +++++++++
.../scheduler/OpportunisticContainerAllocator.java | 347 ++-------
.../src/main/proto/ResourceTracker.proto | 1 +
.../src/main/proto/SCMUploader.proto | 1 +
.../main/proto/collectornodemanager_protocol.proto | 1 +
.../proto/distributed_scheduling_am_protocol.proto | 2 +-
.../src/main/proto/yarn_server_common_protos.proto | 1 +
.../proto/yarn_server_common_service_protos.proto | 1 +
.../main/proto/yarn_server_federation_protos.proto | 1 +
.../TestOpportunisticContainerAllocator.java | 2 +-
.../hadoop-yarn-server-nodemanager/pom.xml | 34 +-
.../yarn/server/nodemanager/NodeManager.java | 3 +-
.../src/main/proto/LocalizationProtocol.proto | 1 +
.../proto/yarn_server_nodemanager_recovery.proto | 1 +
.../yarn_server_nodemanager_service_protos.proto | 1 +
.../logaggregation/TestLogAggregationService.java | 106 +--
.../scheduler/TestDistributedScheduler.java | 4 +-
.../nodemanager/webapp/TestNMWebServices.java | 43 +-
.../hadoop-yarn-server-resourcemanager/pom.xml | 69 +-
.../server/resourcemanager/ClientRMService.java | 6 +-
.../resourcemanager/DefaultAMSProcessor.java | 8 +
.../OpportunisticContainerAllocatorAMService.java | 30 +-
.../yarn/server/resourcemanager/RMAppManager.java | 7 +
.../yarn/server/resourcemanager/RMAuditLogger.java | 53 +-
.../scheduler/MutableConfigurationProvider.java | 2 +
.../scheduler/SchedulerApplicationAttempt.java | 9 +
.../resourcemanager/scheduler/SchedulerUtils.java | 23 +
.../capacity/CapacitySchedulerConfiguration.java | 7 +
.../scheduler/capacity/LeafQueue.java | 8 +-
.../conf/FSSchedulerConfigurationStore.java | 5 +
.../capacity/conf/InMemoryConfigurationStore.java | 5 +
.../capacity/conf/LeveldbConfigurationStore.java | 7 +
.../conf/MutableCSConfigurationProvider.java | 46 +-
.../capacity/conf/YarnConfigurationStore.java | 7 +
.../capacity/conf/ZKConfigurationStore.java | 5 +
.../policy/AbstractComparatorOrderingPolicy.java | 8 +-
.../FifoOrderingPolicyWithExclusivePartitions.java | 144 ++++
.../scheduler/policy/IteratorSelector.java} | 37 +-
.../scheduler/policy/OrderingPolicy.java | 3 +-
.../scheduler/policy/SchedulableEntity.java | 5 +
.../server/resourcemanager/webapp/RMWSConsts.java | 3 +
.../resourcemanager/webapp/RMWebServices.java | 31 +
.../yarn_server_resourcemanager_recovery.proto | 1 +
.../server/resourcemanager/TestRMAuditLogger.java | 21 +-
.../scheduler/TestSchedulerUtils.java | 142 ++++
.../scheduler/capacity/TestCapacityScheduler.java | 11 +-
.../scheduler/capacity/TestLeafQueue.java | 145 +++-
.../conf/TestFSSchedulerConfigurationStore.java | 16 +
.../conf/TestMutableCSConfigurationProvider.java | 8 +
.../capacity/conf/TestZKConfigurationStore.java | 9 +
.../scheduler/policy/MockSchedulableEntity.java | 15 +-
.../scheduler/policy/TestFairOrderingPolicy.java | 12 +-
.../scheduler/policy/TestFifoOrderingPolicy.java | 2 +-
.../TestFifoOrderingPolicyForPendingApps.java | 5 +-
...tFifoOrderingPolicyWithExclusivePartitions.java | 244 ++++++
.../security/TestDelegationTokenRenewer.java | 4 +
.../TestRMWebServicesConfigurationMutation.java | 13 +
.../src/test/proto/test_client_tokens.proto | 1 +
.../hadoop-yarn-server-tests/pom.xml | 31 +-
.../src/test/proto/test_token.proto | 1 +
.../src/site/markdown/NodeManager.md | 8 +-
.../src/site/markdown/ResourceManagerRest.md | 81 ++
pom.ozone.xml | 29 +-
520 files changed, 18064 insertions(+), 4052 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org