You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ae...@apache.org on 2019/07/18 21:25:03 UTC
[hadoop] branch ozone-0.4.1 updated (669c447 -> e1e3701)
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a change to branch ozone-0.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 669c447 HDDS-1785. OOM error in Freon due to the concurrency handling
add 564758a HDDS-1698. Switch to use apache/ozone-runner in the compose/Dockerfile (#979)
add aa9f085 HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.
add 75b1e45 HDDS-1716. Smoketest results are generated with an internal user
add 8b0d1ad HDDS-1616. ManagedChannel references are being leaked in while removing RaftGroup. Contributed by Mukul Kumar Singh. (#1039)
add 91cc197 HDDS-1757. Use ExecutorService in OzoneManagerStateMachine. (#1048)
add 15d82fc YARN-9658. Fix UT failures in TestLeafQueue. Contributed by Tao Yang.
add a2a8be1 YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by Adam Antal.
add 34747c3 HADOOP-16396. Allow authoritative mode on a subdirectory. (#1043)
add 8965ddc HDDS-1731. Implement File CreateFile Request to use Cache and DoubleBuffer. (#1044)
add acd2d52 HDDS-1741. Fix prometheus configuration in ozoneperf example cluster
add 729cb3a HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang.
add 1c254a8 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina.
add c3ca348 HDFS-14620. RBF: Fix 'not a super user' error when disabling a namespace in kerberos with superuser principal. Contributed by luhuachao.
add 928edb2 SUBMARINE-94. Support different docker images for
add c9c3429 HDDS-1758. Add replication and key deletion tests to MiniOzoneChaosCluster. Contributed by Mukul Kumar Singh. (#1049)
add 9c90729 HDFS-13694. Making md5 computing being in parallel with image loading.
add ce91d35 HDDS-1728. Add metrics for leader's latency in ContainerStateMachine. Contributed by Mukul Kumar Singh. (#1022)
add ec851e4 HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL.
add de6b7bc HADOOP-16409. Allow authoritative mode on non-qualified paths. Contributed by Sean Mackrory
add 4632708 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun.
add 738c093 HADOOP-16411. Fix javadoc warnings in hadoop-dynamometer.
add b5d30e4 HDDS-1750. Add block allocation metrics for pipelines in SCM. Contributed by Lokesh Jain. (#1047)
add e5ffb88 HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. (#1050)
add 82d88a8 HDDS-1705. Recon: Add estimatedTotalCount to the response of containers and containers/{id} endpoints. Contributed by Vivek Ratnavel Subramanian.
add 8861573 HDDS-1717. MR Job fails as OMFailoverProxyProvider has dependency hadoop-3.2
add 1dc58c5 HDDS-1742. Merge ozone-perf and ozonetrace example clusters
add 96d0555 HDDS-1718. Increase Ratis Leader election timeout default. Contributed by Aravindan Vijayan & Siddharth Wagle. (#1062)
add 0303072 HDDS-1586. Allow Ozone RPC client to read with topology awareness. Contributed by Sammi Chen.
add ac7a8ac HDDS-1603. Handle Ratis Append Failure in Container State Machine. Contributed by Supratim Deka (#1019)
add 993dc87 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
add e0fa104 HDDS-1748. Error message for 3 way commit failure is not verbose. Contributed by Supratim Deka (#1051)
add 9382488 HDDS-1764. Fix hidden errors in acceptance tests
add bbf5844 HDDS-1525. Mapreduce failure when using Hadoop 2.7.5
add efb9164 HDDS-1778. Fix existing blockade tests. (#1068)
add 90b10a0 HDFS-14622. [Dynamometer] Update XML FsImage parsing logic to ignore non-INodeSection entries to fix an issue caused by the presence of Centralized Cache Management functionality. Contributed by Erik Krogen.
add cdb20ad HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. (#973)
add eccc9a4 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao.
add 6872efc HDDS-1611.[Addendum] Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. (#973)
add c7b5f85 HADOOP-16393. S3Guard init command uses global settings, not those of target bucket.
add 555dabf YARN-9660. Update support documentation for Docker on YARN. Contributed by Peter Bacsko
add ccaa99c HADOOP-16381. The JSON License is included in binary tarball via azure-documentdb:1.16.2. Contributed by Sushil Ks.
add acef5e0 HDDS-1201. Reporting corrupted containers info to SCM (#1032)
add 9cec023 YARN-9625. UI2 - No link to a queue on the Queues page for Fair Scheduler. Contributed by Zoltan Siegl
add fc0656d HADOOP-16418. [Dynamometer] Fix checkstyle and findbugs warnings. Contributed by Erik Krogen.
add 32925d0 HDFS-14640. [Dynamometer] Fix TestDynamometerInfra failure. Contributed by Erik Krogen.
add 5043840 HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen.
add 5747f6c HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell.
add 6a3433b HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
add 8fb5ca3 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
add 9b54dd7 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonathan Hung)
add 0f399b0 HDDS-1784. Missing HostName and IpAddress in the response of register command.
add cdcc205 HDDS-1784. Missing HostName and IpAddress in the response of register command. (#1078)
add 00dd843 HDFS-14466. Add a regression test for HDFS-14323.
add 738fab3 HDDS-1754. getContainerWithPipeline fails with PipelineNotFoundException. Contributed by Supratim Deka (#1081)
add 14c43f8 HDDS-1752. ConcurrentModificationException while handling DeadNodeHandler event. (#1080)
add 9119ed0 HDDS-1384. TestBlockOutputStreamWithFailures is failing
add f9fab9f HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee.
add b15ef7d HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
add 7b8177b HDDS-1791. Update network-tests/src/test/blockade/README.md file
add 190e434 HDDS-1790. Fix checkstyle issues in TestDataScrubber
add c416284 YARN-9235. If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown. Contributed by Antal Balint Steinbach, Adam Antal
add 8b3c679 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko
add 61b0c2b YARN-9337. GPU auto-discovery script runs even when the resource is given by hand. Contributed by Adam Antal
add 557056e YARN-9626. UI2 - Fair scheduler queue apps page issues. Contributed by Zoltan Siegl
add 0bae9e8 HDDS-1735. Create separate unit and integration test executor dev-support script
add 6f25420 fixing ALL the checkstule issues under hadoop-ozone/dev-support/checks
add 987bf3f fix print0 consequences
add 317f2c8 fix rat problem
add ca7b3e8 reenable unit test execution
add 51e7dc3 make author check yetus safe
add 4a70a0d HDDS-1735. Create separate unit and integration test executor dev-support script. Contributed by Elek, Marton.
add 0976f6f HDDS-1766. ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed by Mukul Kumar Singh. (#1072)
add e98adb0 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.
add 30a8f84 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He.
add 91ce09e YARN-9360. Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource. Contributed by Peter Bacsko
add be784de SUBMARINE-62. PS_LAUNCH_CMD CLI description is wrong in RunJobCli. Contributed by Adam Antal
add 18ee109 YARN-9127. Create more tests to verify GpuDeviceInformationParser. Contributed by Peter Bacsko
add 5446308 YARN-9326. Fair Scheduler configuration defaults are not documented in case of min and maxResources. Contributed by Adam Antal
add 61bbdee HDDS-1800. Result of author check is inverted
add 64d4abf HDFS-14593. RBF: Implement deletion feature for expired records in State Store. Contributed by Takanobu Asanuma.
add 68f53f9 HDDS-1763. Use vendor neutral s3 logo in ozone doc. Contributed by Elek, Marton.
add 47345e5 Merge pull request #1058 from elek/HDDS-1763
add d3a5abe HDDS-1668. Add liveness probe to the example k8s resources files
add 1411513 Merge pull request #944 from elek/HDDS-1668
add 585f4d5 HDDS-1761. Fix class hierarchy for KeyRequest and FileRequest classes. (#1052)
add ef66e49 HDDS-1666. Issue in openKey when allocating block. (#943)
add f77d54c HDFS-14642. processMisReplicatedBlocks does not return correct processed count. Contributed by Stephen O'Donnell.
add 395cb3c HDDS-1736. Cleanup 2phase old HA code for Key requests. (#1038)
add 7a93be0 YARN-9645. Fix Invalid event FINISHED_CONTAINERS_PULLED_BY_AM at NEW on NM restart. Contributed by Bilwa S T.
add 1502e6a HDDS-1756. DeleteContainerCommandHandler fails with NPE. Contributed by Nanda kumar(#1095).
add d21eccf HDDS-1492. Generated chunk size name too long. Contributed by Shashikannt Banerjee. (#1084)
add c5e3ab5 HDDS-1793. Acceptance test of ozone-topology cluster is failing
add 493b0b5 HDDS-1787. NPE thrown while trying to find DN closest to client. Contributed by Sammi Chen. (#1094)
add d429d74 HDDS-1544. Support default Acls for volume, bucket, keys and prefix. Contributed by Ajay Kumar, Xiaoyu Yao.
add 71c21f0 HDDS-1813. Fix false warning from ozones3 acceptance test. Contributed by Xiaoyu Yao. (#1107)
add 7adc4a3 HDDS-1775. Make OM KeyDeletingService compatible with HA model (#1063)
add 5672efa HADOOP-15729. [s3a] Allow core threads to time out. (#1075)
add 5915c90 YARN-9646. DistributedShell tests failed to bind to a local host name. (Contributed by Ray Yang)
add 85d9111 YARN-9682. Wrong log message when finalizing the upgrade. Contributed by kyungwan nam.
add ee3115f HDFS-9913. DistCp to add -useTrash to move deleted files to Trash.
add 19a0018 Revert "HDFS-9913. DistCp to add -useTrash to move deleted files to Trash."
add b4466a3 HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
add 256fcc1 HDDS-1785. OOM error in Freon due to the concurrency handling
add c58e11b HADOOP-16383. Pass ITtlTimeProvider instance in initialize method in MetadataStore interface. Contributed by Gabor Bota. (#1009)
add eb0849b HDDS-1801. Make Topology Aware Replication/Read non-default for ozone 0.4.1. Contributed by Xiaoyu Yao.
new e1e3701 Merge branch 'trunk' into ozone-0.4.1
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/net/NetworkTopology.java | 9 +-
.../org/apache/hadoop/security/token/Token.java | 29 +-
.../service/launcher/ServiceLaunchException.java | 14 +
.../hadoop/service/launcher/ServiceLauncher.java | 70 +-
.../apache/hadoop/util/ShutdownHookManager.java | 21 +-
.../hadoop-common/src/main/proto/Security.proto | 1 -
.../src/main/resources/core-default.xml | 2 +-
.../AbstractContractRootDirectoryTest.java | 71 +-
.../hadoop/fs/contract/ContractTestUtils.java | 4 +-
.../hadoop/util/TestShutdownHookManager.java | 29 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 38 +-
.../hadoop/hdds/scm/XceiverClientManager.java | 68 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 7 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 4 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 25 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 28 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 +-
.../hadoop/hdds/scm/net/NetworkTopology.java | 4 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 53 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 79 ++-
.../scm/protocol/ScmBlockLocationProtocol.java | 9 +
...lockLocationProtocolClientSideTranslatorPB.java | 34 +
.../hdds/scm/storage/ContainerProtocolCalls.java | 4 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
...lockLocationProtocolServerSideTranslatorPB.java | 26 +
.../src/main/proto/ScmBlockLocationProtocol.proto | 16 +
hadoop-hdds/common/src/main/proto/hdds.proto | 5 +-
.../common/src/main/resources/ozone-default.xml | 12 +-
.../hdds/scm/net/TestNetworkTopologyImpl.java | 35 +-
.../container/common/interfaces/Container.java | 4 +-
.../ozone/container/common/interfaces/Handler.java | 5 +
.../container/common/report/ReportManager.java | 10 +
.../states/endpoint/HeartbeatEndpointTask.java | 6 +-
.../common/transport/server/XceiverServerGrpc.java | 6 +
.../common/transport/server/ratis/CSMMetrics.java | 30 +
.../server/ratis/ContainerStateMachine.java | 30 +-
.../transport/server/ratis/XceiverServerRatis.java | 30 +-
.../container/keyvalue/KeyValueContainer.java | 20 +-
.../container/keyvalue/KeyValueContainerCheck.java | 46 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 18 +-
.../container/ozoneimpl/ContainerController.java | 16 +-
.../container/ozoneimpl/ContainerScrubber.java | 31 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 1 +
.../proto/StorageContainerDatanodeProtocol.proto | 1 +
.../keyvalue/TestKeyValueContainerCheck.java | 11 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 1 +
.../IncrementalContainerReportHandler.java | 2 +
.../ContainerPlacementPolicyFactory.java | 6 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 17 +-
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 2 +
.../hdds/scm/pipeline/SCMPipelineManager.java | 10 +-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 56 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 37 +
.../hadoop/hdds/scm/server/SCMConfigurator.java | 19 +
.../hdds/scm/server/StorageContainerManager.java | 14 +-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 4 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 14 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 10 +
.../scm/server/TestSCMBlockProtocolServer.java | 153 ++++
.../java/org/apache/hadoop/hdfs/DFSUtilClient.java | 20 +
.../apache/hadoop/hdfs/DistributedFileSystem.java | 8 +-
.../datatransfer/sasl/SaslDataTransferClient.java | 55 +-
.../hadoop/hdfs/protocolPB/PBHelperClient.java | 12 +-
.../security/token/block/BlockTokenIdentifier.java | 32 +
.../hadoop-hdfs-client/src/main/proto/hdfs.proto | 1 +
.../server/federation/router/RBFConfigKeys.java | 8 +
.../federation/router/RouterPermissionChecker.java | 2 +-
.../server/federation/store/CachedRecordStore.java | 22 +-
.../server/federation/store/StateStoreService.java | 16 +-
.../federation/store/records/BaseRecord.java | 46 +-
.../federation/store/records/MembershipState.java | 21 +
.../federation/store/records/QueryResult.java | 3 +-
.../federation/store/records/RouterState.java | 17 +
.../records/impl/pb/MembershipStatePBImpl.java | 4 +-
.../store/records/impl/pb/RouterStatePBImpl.java | 4 +-
.../src/main/resources/hdfs-rbf-default.xml | 20 +
.../server/federation/router/TestRouterAdmin.java | 52 +-
.../store/TestStateStoreMembershipState.java | 100 ++-
.../store/TestStateStoreRouterState.java | 61 +-
.../store/driver/TestStateStoreDriverBase.java | 5 +-
.../datatransfer/sasl/SaslDataTransferServer.java | 28 +-
.../token/block/BlockTokenSecretManager.java | 59 +-
.../hdfs/server/blockmanagement/BlockManager.java | 9 +-
.../blockmanagement/BlockManagerSafeMode.java | 22 +-
.../blockmanagement/DatanodeAdminManager.java | 151 ++--
.../apache/hadoop/hdfs/server/datanode/DNConf.java | 6 +-
.../{MemoryCacheStats.java => CacheStats.java} | 6 +-
.../datanode/fsdataset/impl/FsDatasetCache.java | 54 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 4 +-
.../fsdataset/impl/MappableBlockLoader.java | 3 +-
.../fsdataset/impl/MemoryMappableBlockLoader.java | 8 +-
.../impl/NativePmemMappableBlockLoader.java | 5 +-
.../fsdataset/impl/PmemMappableBlockLoader.java | 9 +-
.../server/namenode/FSImageFormatProtobuf.java | 65 +-
.../hdfs/server/namenode/FSImageSerialization.java | 4 +-
.../hdfs/server/namenode/INodeReference.java | 17 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 47 +-
.../hadoop/hdfs/server/namenode/QuotaCounts.java | 153 +++-
.../web/resources/NamenodeWebHdfsMethods.java | 5 +-
.../apache/hadoop/hdfs/util/ConstEnumCounters.java | 98 +++
.../org/apache/hadoop/hdfs/util/EnumCounters.java | 29 +-
.../site/markdown/CentralizedCacheManagement.md | 18 +-
.../hadoop/hdfs/TestBlockTokenWrappingQOP.java | 43 +-
.../org/apache/hadoop/hdfs/TestDecommission.java | 52 ++
.../java/org/apache/hadoop/hdfs/TestSafeMode.java | 10 +-
.../blockmanagement/TestBlockManagerSafeMode.java | 25 +
.../TestBlocksWithNotEnoughRacks.java | 2 +-
.../impl/TestCacheByPmemMappableBlockLoader.java | 14 +-
.../fsdataset/impl/TestFsDatasetCache.java | 2 +-
.../hdfs/server/namenode/TestCacheDirectives.java | 64 ++
.../hdfs/server/namenode/TestQuotaCounts.java | 135 ++++
.../hdfs/server/namenode/ha/TestHASafeMode.java | 29 +-
.../TestGetContentSummaryWithSnapshot.java | 33 +-
.../org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java | 14 +
.../ozone/client/io/BlockOutputStreamEntry.java | 2 +-
hadoop-ozone/common/src/main/bin/ozone | 2 +
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 13 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 4 +-
.../ozone/om/protocol/OzoneManagerHAProtocol.java | 40 --
.../src/main/proto/OzoneManagerProtocol.proto | 24 +-
.../compose/ozone-mr/hadoop27/{.env => .env~HEAD} | 0
.../compose/ozone-mr/hadoop27/{.env => .env~trunk} | 0
.../compose/ozone-mr/hadoop31/{.env => .env~HEAD} | 0
.../compose/ozone-mr/hadoop31/{.env => .env~trunk} | 0
.../{ozone-net-topology => ozone-topology}/.env | 0
.../docker-compose.yaml | 16 +-
.../docker-config | 8 +-
.../network-config | 0
.../{ozone-net-topology => ozone-topology}/test.sh | 0
.../dist/src/main/smoketest/s3/bucketcreate.robot | 2 +-
.../ozone/{constants.py => constants.py~HEAD} | 0
.../ozone/{constants.py => constants.py~trunk} | 0
.../ozone/{exceptions.py => exceptions.py~HEAD} | 0
.../ozone/{exceptions.py => exceptions.py~trunk} | 0
.../hdds/scm/pipeline/TestPipelineClose.java | 81 +++
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 59 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 19 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 5 +
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 2 +-
.../hadoop/ozone/TestStorageContainerManager.java | 17 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 83 ++-
.../client/rpc/TestOzoneRpcClientWithRatis.java | 96 ++-
.../transport/server/ratis/TestCSMMetrics.java | 6 +
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 215 ++++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 132 +++-
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 138 ++++
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 4 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 175 ++++-
.../hadoop/ozone/scm/node/TestQueryNode.java | 16 +-
.../ozone/scm/pipeline/TestSCMPipelineMetrics.java | 35 +
.../apache/hadoop/ozone/om/KeyDeletingService.java | 123 +++-
.../org/apache/hadoop/ozone/om/KeyManager.java | 36 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 173 ++---
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 9 -
.../org/apache/hadoop/ozone/om/OzoneManager.java | 126 ++--
.../apache/hadoop/ozone/om/fs/OzoneManagerFS.java | 11 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 56 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 6 +
.../hadoop/ozone/om/request/OMClientRequest.java | 6 +-
.../om/request/file/OMDirectoryCreateRequest.java | 77 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 349 +++++++++
.../ozone/om/request/file/OMFileRequest.java | 116 +++
.../om/request/key/OMAllocateBlockRequest.java | 6 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 225 +-----
.../ozone/om/request/key/OMKeyDeleteRequest.java | 30 +-
.../ozone/om/request/key/OMKeyPurgeRequest.java | 48 ++
.../ozone/om/request/key/OMKeyRenameRequest.java | 15 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 284 +++++++-
.../om/response/file/OMFileCreateResponse.java | 35 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 9 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 14 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 36 +
.../OzoneManagerHARequestHandlerImpl.java | 2 +
.../protocolPB/OzoneManagerRequestHandler.java | 34 +-
.../ozone/om/ScmBlockLocationTestIngClient.java | 6 +
.../ozone/om/request/TestOMRequestUtils.java | 19 +
.../om/request/file/TestOMFileCreateRequest.java | 371 ++++++++++
.../key/TestOMKeyPurgeRequestAndResponse.java | 146 ++++
.../om/response/key/TestOMKeyDeleteResponse.java | 22 +-
hadoop-project/src/site/site.xml | 1 +
.../submarine/client/cli/runjob/RunJobCli.java | 149 ++--
.../hadoop-submarine-tony-runtime/pom.xml | 2 +-
hadoop-tools/hadoop-aws/pom.xml | 14 +-
.../java/org/apache/hadoop/fs/s3a/Constants.java | 4 +
.../java/org/apache/hadoop/fs/s3a/Listing.java | 11 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 50 +-
.../apache/hadoop/fs/s3a/S3ALocatedFileStatus.java | 15 +
.../apache/hadoop/fs/s3a/WriteOperationHelper.java | 2 +-
.../hadoop/fs/s3a/commit/CommitConstants.java | 2 +-
.../hadoop/fs/s3a/commit/CommitOperations.java | 2 +
.../commit/staging/DirectoryStagingCommitter.java | 34 +-
.../fs/s3a/commit/staging/StagingCommitter.java | 7 +-
.../fs/s3a/impl/MultiObjectDeleteSupport.java | 2 +-
.../s3guard/AbstractS3GuardDynamoDBDiagnostic.java | 223 ++++++
.../hadoop/fs/s3a/s3guard/BulkOperationState.java | 6 +
.../fs/s3a/s3guard/DelayedUpdateRenameTracker.java | 12 +-
.../fs/s3a/s3guard/DumpS3GuardDynamoTable.java | 787 +++++++++++++++++++++
.../fs/s3a/s3guard/DynamoDBMetadataStore.java | 428 ++++++++---
.../hadoop/fs/s3a/s3guard/LocalMetadataStore.java | 39 +-
.../hadoop/fs/s3a/s3guard/MetadataStore.java | 44 +-
.../hadoop/fs/s3a/s3guard/NullMetadataStore.java | 16 +-
.../s3guard/PathMetadataDynamoDBTranslation.java | 39 +-
.../fs/s3a/s3guard/PathOrderComparators.java | 4 +-
.../fs/s3a/s3guard/ProgressiveRenameTracker.java | 16 +-
.../fs/s3a/s3guard/PurgeS3GuardDynamoTable.java | 248 +++++++
.../hadoop/fs/s3a/s3guard/RenameTracker.java | 2 +-
.../org/apache/hadoop/fs/s3a/s3guard/S3Guard.java | 48 +-
.../hadoop/fs/s3a/s3guard/S3GuardTableAccess.java | 241 +++++++
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 37 +-
.../site/markdown/tools/hadoop-aws/committers.md | 11 +-
.../src/site/markdown/tools/hadoop-aws/s3guard.md | 22 +-
.../src/site/markdown/tools/hadoop-aws/testing.md | 99 ++-
.../fs/contract/s3a/ITestS3AContractRootDir.java | 18 +-
.../apache/hadoop/fs/s3a/AbstractS3ATestBase.java | 7 +
.../hadoop/fs/s3a/ITestAuthoritativePath.java | 305 ++++++++
.../s3a/ITestS3AMetadataPersistenceException.java | 4 +-
.../hadoop/fs/s3a/ITestS3GuardEmptyDirs.java | 44 ++
.../hadoop/fs/s3a/ITestS3GuardListConsistency.java | 119 ++--
.../fs/s3a/ITestS3GuardOutOfBandOperations.java | 51 +-
.../apache/hadoop/fs/s3a/MockS3AFileSystem.java | 11 +-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 51 ++
.../hadoop/fs/s3a/commit/AbstractCommitITest.java | 89 +--
.../fs/s3a/commit/AbstractITCommitMRJob.java | 13 +-
.../fs/s3a/commit/staging/StagingTestBase.java | 34 +-
.../s3a/commit/staging/TestStagingCommitter.java | 4 +-
.../TestStagingDirectoryOutputCommitter.java | 90 ++-
.../staging/TestStagingPartitionedTaskCommit.java | 38 +-
.../integration/ITestDirectoryCommitProtocol.java | 34 +
.../commit/terasort/AbstractCommitTerasortIT.java | 23 +-
.../fs/s3a/impl/ITestPartialRenamesDeletes.java | 42 --
.../fs/s3a/impl/TestPartialDeleteFailures.java | 18 +-
.../s3a/s3guard/AbstractS3GuardToolTestBase.java | 15 +-
.../fs/s3a/s3guard/ITestDynamoDBMetadataStore.java | 306 +++++---
.../s3guard/ITestDynamoDBMetadataStoreScale.java | 23 +-
.../fs/s3a/s3guard/ITestS3GuardConcurrentOps.java | 4 +-
.../s3a/s3guard/ITestS3GuardDDBRootOperations.java | 269 +++++++
.../fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java | 27 +-
.../fs/s3a/s3guard/MetadataStoreTestBase.java | 95 ++-
.../fs/s3a/s3guard/TestDynamoDBMiscOperations.java | 4 +-
.../scale/AbstractITestS3AMetadataStoreScale.java | 8 +-
.../fs/s3a/scale/ITestLocalMetadataStoreScale.java | 3 +-
.../hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java | 175 +++--
.../hadoop-aws/src/test/resources/log4j.properties | 2 +
.../dynamometer/blockgenerator/BlockInfo.java | 50 +-
.../GenerateDNBlockInfosReducer.java | 2 +-
.../dynamometer/blockgenerator/XMLParser.java | 30 +-
.../dynamometer/blockgenerator/TestXMLParser.java | 31 +-
.../hadoop-dynamometer-infra/pom.xml | 10 +
.../src/main/bash/create-slim-hadoop-tar.sh | 2 +-
.../src/main/bash/upload-fsimage.sh | 4 +-
.../apache/hadoop/tools/dynamometer/AMOptions.java | 9 +-
.../tools/dynamometer/ApplicationMaster.java | 16 +-
.../apache/hadoop/tools/dynamometer/Client.java | 34 +-
.../hadoop/tools/dynamometer/DynoInfraUtils.java | 11 +-
.../tools/dynamometer/SimulatedDataNodes.java | 13 +-
.../src/main/resources/start-component.sh | 9 +-
.../tools/dynamometer/TestDynamometerInfra.java | 169 +++--
.../src/main/bash/parse-start-timestamp.sh | 4 +-
.../workloadgenerator/CreateFileMapper.java | 4 +-
.../workloadgenerator/WorkloadDriver.java | 14 +-
.../workloadgenerator/WorkloadMapper.java | 6 +
.../audit/AuditCommandParser.java | 2 +
.../audit/AuditLogDirectParser.java | 4 +-
.../audit/AuditLogHiveTableParser.java | 4 +-
.../workloadgenerator/audit/AuditReplayMapper.java | 8 +-
.../src/site/markdown/Dynamometer.md | 299 ++++++++
.../src/site/resources/css/site.css | 0
.../images/dynamometer-architecture-infra.png | Bin 0 -> 123874 bytes
.../images/dynamometer-architecture-replay.png | Bin 0 -> 159507 bytes
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 8 +
.../distributedshell/TestDistributedShell.java | 66 +-
.../hadoop/yarn/service/client/ServiceClient.java | 2 +-
.../src/main/resources/yarn-default.xml | 17 +-
.../resource-types-error-redefine-fpga-unit.xml | 45 ++
.../resource-types-error-redefine-gpu-unit.xml | 45 ++
.../yarn/server/nodemanager/NodeManager.java | 1 +
.../resources/numa/NumaResourceAllocation.java | 59 +-
.../resources/numa/NumaResourceAllocator.java | 34 +-
.../logaggregation/LogAggregationService.java | 79 ++-
.../resourceplugin/gpu/GpuDiscoverer.java | 62 +-
.../resourceplugin/gpu/GpuResourcePlugin.java | 22 +
.../recovery/NMLeveldbStateStoreService.java | 5 +-
.../webapp/dao/gpu/GpuDeviceInformation.java | 18 +-
.../webapp/dao/gpu/GpuDeviceInformationParser.java | 48 +-
.../webapp/dao/gpu/PerGpuDeviceInformation.java | 1 -
.../webapp/dao/gpu/PerGpuMemoryUsage.java | 7 +-
.../yarn/server/nodemanager/TestNodeManager.java | 28 +
.../logaggregation/TestLogAggregationService.java | 39 +
.../resourceplugin/gpu/TestGpuDiscoverer.java | 19 +-
.../resourceplugin/gpu/TestGpuResourcePlugin.java | 54 ++
.../recovery/TestNMLeveldbStateStoreService.java | 52 +-
.../dao/gpu/TestGpuDeviceInformationParser.java | 157 +++-
.../test/resources/nvidia-smi-output-excerpt.xml | 71 ++
.../resources/nvidia-smi-output-missing-tags.xml | 28 +
.../resources/nvidia-smi-output-missing-tags2.xml | 61 ++
.../test/resources/nvidia-smi-sample-output.xml | 547 ++++++++++++++
.../test/resources/nvidia-smi-sample-xml-output | 547 --------------
.../server/resourcemanager/ResourceManager.java | 1 +
.../server/resourcemanager/rmnode/RMNodeImpl.java | 8 +
.../resourcemanager/scheduler/QueueMetrics.java | 34 +-
.../scheduler/fair/FSLeafQueue.java | 33 +-
.../resourcemanager/TestRMNodeTransitions.java | 18 +
.../yarn/server/resourcemanager/TestRMRestart.java | 5 +-
.../resourcemanager/TestResourceManager.java | 30 +-
.../scheduler/capacity/TestLeafQueue.java | 1 +
.../apache/hadoop/yarn/server/MiniYARNCluster.java | 20 +-
.../pom.xml | 40 +-
.../documentstore/DocumentStoreUtils.java | 27 +-
.../document/entity/TimelineEventSubDoc.java | 7 +-
.../document/entity/TimelineMetricSubDoc.java | 6 -
.../cosmosdb/CosmosDBDocumentStoreReader.java | 115 +--
.../cosmosdb/CosmosDBDocumentStoreWriter.java | 203 ++++--
.../cosmosdb/TestCosmosDBDocumentStoreReader.java | 67 ++
.../MockedCosmosDBDocumentStoreWriter.java | 47 ++
.../cosmosdb/TestCosmosDBDocumentStoreWriter.java | 96 +++
.../src/site/markdown/DockerContainers.md | 86 +++
.../src/site/markdown/FairScheduler.md | 14 +-
.../yarn-queue/fair-queue-conf-table.hbs | 2 +-
.../templates/components/yarn-queue/fair-queue.hbs | 12 +
.../src/main/webapp/app/templates/yarn-queue.hbs | 44 +-
321 files changed, 11603 insertions(+), 3351 deletions(-)
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMBlockProtocolServer.java
rename hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/{MemoryCacheStats.java => CacheStats.java} (97%)
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/ConstEnumCounters.java
create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaCounts.java
copy hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop27/{.env => .env~HEAD} (100%)
rename hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop27/{.env => .env~trunk} (100%)
copy hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/{.env => .env~HEAD} (100%)
rename hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/{.env => .env~trunk} (100%)
rename hadoop-ozone/dist/src/main/compose/{ozone-net-topology => ozone-topology}/.env (100%)
rename hadoop-ozone/dist/src/main/compose/{ozone-net-topology => ozone-topology}/docker-compose.yaml (92%)
rename hadoop-ozone/dist/src/main/compose/{ozone-net-topology => ozone-topology}/docker-config (91%)
rename hadoop-ozone/dist/src/main/compose/{ozone-net-topology => ozone-topology}/network-config (100%)
rename hadoop-ozone/dist/src/main/compose/{ozone-net-topology => ozone-topology}/test.sh (100%)
copy hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/ozone/{constants.py => constants.py~HEAD} (100%)
rename hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/ozone/{constants.py => constants.py~trunk} (100%)
copy hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/ozone/{exceptions.py => exceptions.py~HEAD} (100%)
rename hadoop-ozone/fault-injection-test/network-tests/src/test/blockade/ozone/{exceptions.py => exceptions.py~trunk} (100%)
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scrubber/TestDataScrubber.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyPurging.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
copy hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestLocalMetadataStoreScale.java => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMFileCreateResponse.java (54%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMFileCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyPurgeRequestAndResponse.java
create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardDynamoDBDiagnostic.java
create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DumpS3GuardDynamoTable.java
create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PurgeS3GuardDynamoTable.java
create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTableAccess.java
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestAuthoritativePath.java
create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardDDBRootOperations.java
create mode 100644 hadoop-tools/hadoop-dynamometer/src/site/markdown/Dynamometer.md
copy {hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site => hadoop-tools/hadoop-dynamometer}/src/site/resources/css/site.css (100%)
create mode 100644 hadoop-tools/hadoop-dynamometer/src/site/resources/images/dynamometer-architecture-infra.png
create mode 100644 hadoop-tools/hadoop-dynamometer/src/site/resources/images/dynamometer-architecture-replay.png
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/resource-types/resource-types-error-redefine-fpga-unit.xml
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/resource-types/resource-types-error-redefine-gpu-unit.xml
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuResourcePlugin.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-xml-output
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/reader/cosmosdb/TestCosmosDBDocumentStoreReader.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/writer/cosmosdb/MockedCosmosDBDocumentStoreWriter.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/writer/cosmosdb/TestCosmosDBDocumentStoreWriter.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/01: Merge branch 'trunk' into ozone-0.4.1
Posted by ae...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a commit to branch ozone-0.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit e1e3701f3582693621974913eb54a610079acb5d
Merge: 669c447 eb0849b
Author: Anu Engineer <ae...@apache.org>
AuthorDate: Thu Jul 18 13:42:37 2019 -0700
Merge branch 'trunk' into ozone-0.4.1
.../org/apache/hadoop/net/NetworkTopology.java | 9 +-
.../org/apache/hadoop/security/token/Token.java | 29 +-
.../service/launcher/ServiceLaunchException.java | 14 +
.../hadoop/service/launcher/ServiceLauncher.java | 70 +-
.../apache/hadoop/util/ShutdownHookManager.java | 21 +-
.../hadoop-common/src/main/proto/Security.proto | 1 -
.../src/main/resources/core-default.xml | 2 +-
.../AbstractContractRootDirectoryTest.java | 71 +-
.../hadoop/fs/contract/ContractTestUtils.java | 4 +-
.../hadoop/util/TestShutdownHookManager.java | 29 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 38 +-
.../hadoop/hdds/scm/XceiverClientManager.java | 68 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 7 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 4 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 25 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 28 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 +-
.../hadoop/hdds/scm/net/NetworkTopology.java | 4 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 53 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 79 ++-
.../scm/protocol/ScmBlockLocationProtocol.java | 9 +
...lockLocationProtocolClientSideTranslatorPB.java | 34 +
.../hdds/scm/storage/ContainerProtocolCalls.java | 4 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
...lockLocationProtocolServerSideTranslatorPB.java | 26 +
.../src/main/proto/ScmBlockLocationProtocol.proto | 16 +
hadoop-hdds/common/src/main/proto/hdds.proto | 5 +-
.../common/src/main/resources/ozone-default.xml | 12 +-
.../hdds/scm/net/TestNetworkTopologyImpl.java | 35 +-
.../container/common/interfaces/Container.java | 4 +-
.../ozone/container/common/interfaces/Handler.java | 5 +
.../container/common/report/ReportManager.java | 10 +
.../states/endpoint/HeartbeatEndpointTask.java | 6 +-
.../common/transport/server/XceiverServerGrpc.java | 6 +
.../common/transport/server/ratis/CSMMetrics.java | 30 +
.../server/ratis/ContainerStateMachine.java | 30 +-
.../transport/server/ratis/XceiverServerRatis.java | 30 +-
.../container/keyvalue/KeyValueContainer.java | 20 +-
.../container/keyvalue/KeyValueContainerCheck.java | 46 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 18 +-
.../container/ozoneimpl/ContainerController.java | 16 +-
.../container/ozoneimpl/ContainerScrubber.java | 31 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 1 +
.../proto/StorageContainerDatanodeProtocol.proto | 1 +
.../keyvalue/TestKeyValueContainerCheck.java | 11 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 1 +
.../IncrementalContainerReportHandler.java | 2 +
.../ContainerPlacementPolicyFactory.java | 6 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 17 +-
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 2 +
.../hdds/scm/pipeline/SCMPipelineManager.java | 10 +-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 56 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 37 +
.../hadoop/hdds/scm/server/SCMConfigurator.java | 19 +
.../hdds/scm/server/StorageContainerManager.java | 14 +-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 4 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 14 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 10 +
.../scm/server/TestSCMBlockProtocolServer.java | 153 ++++
.../java/org/apache/hadoop/hdfs/DFSUtilClient.java | 20 +
.../apache/hadoop/hdfs/DistributedFileSystem.java | 8 +-
.../datatransfer/sasl/SaslDataTransferClient.java | 55 +-
.../hadoop/hdfs/protocolPB/PBHelperClient.java | 12 +-
.../security/token/block/BlockTokenIdentifier.java | 32 +
.../hadoop-hdfs-client/src/main/proto/hdfs.proto | 1 +
.../server/federation/router/RBFConfigKeys.java | 8 +
.../federation/router/RouterPermissionChecker.java | 2 +-
.../server/federation/store/CachedRecordStore.java | 22 +-
.../server/federation/store/StateStoreService.java | 16 +-
.../federation/store/records/BaseRecord.java | 46 +-
.../federation/store/records/MembershipState.java | 21 +
.../federation/store/records/QueryResult.java | 3 +-
.../federation/store/records/RouterState.java | 17 +
.../records/impl/pb/MembershipStatePBImpl.java | 4 +-
.../store/records/impl/pb/RouterStatePBImpl.java | 4 +-
.../src/main/resources/hdfs-rbf-default.xml | 20 +
.../server/federation/router/TestRouterAdmin.java | 52 +-
.../store/TestStateStoreMembershipState.java | 100 ++-
.../store/TestStateStoreRouterState.java | 61 +-
.../store/driver/TestStateStoreDriverBase.java | 5 +-
.../datatransfer/sasl/SaslDataTransferServer.java | 28 +-
.../token/block/BlockTokenSecretManager.java | 59 +-
.../hdfs/server/blockmanagement/BlockManager.java | 9 +-
.../blockmanagement/BlockManagerSafeMode.java | 22 +-
.../blockmanagement/DatanodeAdminManager.java | 151 ++--
.../apache/hadoop/hdfs/server/datanode/DNConf.java | 6 +-
.../{MemoryCacheStats.java => CacheStats.java} | 6 +-
.../datanode/fsdataset/impl/FsDatasetCache.java | 54 +-
.../datanode/fsdataset/impl/FsDatasetImpl.java | 4 +-
.../fsdataset/impl/MappableBlockLoader.java | 3 +-
.../fsdataset/impl/MemoryMappableBlockLoader.java | 8 +-
.../impl/NativePmemMappableBlockLoader.java | 5 +-
.../fsdataset/impl/PmemMappableBlockLoader.java | 9 +-
.../server/namenode/FSImageFormatProtobuf.java | 65 +-
.../hdfs/server/namenode/FSImageSerialization.java | 4 +-
.../hdfs/server/namenode/INodeReference.java | 17 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 47 +-
.../hadoop/hdfs/server/namenode/QuotaCounts.java | 153 +++-
.../web/resources/NamenodeWebHdfsMethods.java | 5 +-
.../apache/hadoop/hdfs/util/ConstEnumCounters.java | 98 +++
.../org/apache/hadoop/hdfs/util/EnumCounters.java | 29 +-
.../site/markdown/CentralizedCacheManagement.md | 18 +-
.../hadoop/hdfs/TestBlockTokenWrappingQOP.java | 43 +-
.../org/apache/hadoop/hdfs/TestDecommission.java | 52 ++
.../java/org/apache/hadoop/hdfs/TestSafeMode.java | 10 +-
.../blockmanagement/TestBlockManagerSafeMode.java | 25 +
.../TestBlocksWithNotEnoughRacks.java | 2 +-
.../impl/TestCacheByPmemMappableBlockLoader.java | 14 +-
.../fsdataset/impl/TestFsDatasetCache.java | 2 +-
.../hdfs/server/namenode/TestCacheDirectives.java | 64 ++
.../hdfs/server/namenode/TestQuotaCounts.java | 135 ++++
.../hdfs/server/namenode/ha/TestHASafeMode.java | 29 +-
.../TestGetContentSummaryWithSnapshot.java | 33 +-
.../org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java | 14 +
.../ozone/client/io/BlockOutputStreamEntry.java | 2 +-
hadoop-ozone/common/src/main/bin/ozone | 2 +
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 13 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 4 +-
.../ozone/om/protocol/OzoneManagerHAProtocol.java | 40 --
.../src/main/proto/OzoneManagerProtocol.proto | 24 +-
.../compose/ozone-mr/hadoop27/{.env => .env~HEAD} | 0
.../compose/ozone-mr/hadoop27/{.env => .env~trunk} | 0
.../compose/ozone-mr/hadoop31/{.env => .env~HEAD} | 0
.../compose/ozone-mr/hadoop31/{.env => .env~trunk} | 0
.../{ozone-net-topology => ozone-topology}/.env | 0
.../docker-compose.yaml | 16 +-
.../docker-config | 8 +-
.../network-config | 0
.../{ozone-net-topology => ozone-topology}/test.sh | 0
.../dist/src/main/smoketest/s3/bucketcreate.robot | 2 +-
.../ozone/{constants.py => constants.py~HEAD} | 0
.../ozone/{constants.py => constants.py~trunk} | 0
.../ozone/{exceptions.py => exceptions.py~HEAD} | 0
.../ozone/{exceptions.py => exceptions.py~trunk} | 0
.../hdds/scm/pipeline/TestPipelineClose.java | 81 +++
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 59 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 19 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 5 +
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 2 +-
.../hadoop/ozone/TestStorageContainerManager.java | 17 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 83 ++-
.../client/rpc/TestOzoneRpcClientWithRatis.java | 96 ++-
.../transport/server/ratis/TestCSMMetrics.java | 6 +
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 215 ++++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 132 +++-
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 138 ++++
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 4 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 175 ++++-
.../hadoop/ozone/scm/node/TestQueryNode.java | 16 +-
.../ozone/scm/pipeline/TestSCMPipelineMetrics.java | 35 +
.../apache/hadoop/ozone/om/KeyDeletingService.java | 123 +++-
.../org/apache/hadoop/ozone/om/KeyManager.java | 36 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 173 ++---
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 9 -
.../org/apache/hadoop/ozone/om/OzoneManager.java | 126 ++--
.../apache/hadoop/ozone/om/fs/OzoneManagerFS.java | 11 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 56 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 6 +
.../hadoop/ozone/om/request/OMClientRequest.java | 6 +-
.../om/request/file/OMDirectoryCreateRequest.java | 77 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 349 +++++++++
.../ozone/om/request/file/OMFileRequest.java | 116 +++
.../om/request/key/OMAllocateBlockRequest.java | 6 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 225 +-----
.../ozone/om/request/key/OMKeyDeleteRequest.java | 30 +-
.../ozone/om/request/key/OMKeyPurgeRequest.java | 48 ++
.../ozone/om/request/key/OMKeyRenameRequest.java | 15 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 284 +++++++-
.../om/response/file/OMFileCreateResponse.java | 35 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 9 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 14 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 36 +
.../OzoneManagerHARequestHandlerImpl.java | 2 +
.../protocolPB/OzoneManagerRequestHandler.java | 34 +-
.../ozone/om/ScmBlockLocationTestIngClient.java | 6 +
.../ozone/om/request/TestOMRequestUtils.java | 19 +
.../om/request/file/TestOMFileCreateRequest.java | 371 ++++++++++
.../key/TestOMKeyPurgeRequestAndResponse.java | 146 ++++
.../om/response/key/TestOMKeyDeleteResponse.java | 22 +-
hadoop-project/src/site/site.xml | 1 +
.../submarine/client/cli/runjob/RunJobCli.java | 149 ++--
.../hadoop-submarine-tony-runtime/pom.xml | 2 +-
hadoop-tools/hadoop-aws/pom.xml | 14 +-
.../java/org/apache/hadoop/fs/s3a/Constants.java | 4 +
.../java/org/apache/hadoop/fs/s3a/Listing.java | 11 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 50 +-
.../apache/hadoop/fs/s3a/S3ALocatedFileStatus.java | 15 +
.../apache/hadoop/fs/s3a/WriteOperationHelper.java | 2 +-
.../hadoop/fs/s3a/commit/CommitConstants.java | 2 +-
.../hadoop/fs/s3a/commit/CommitOperations.java | 2 +
.../commit/staging/DirectoryStagingCommitter.java | 34 +-
.../fs/s3a/commit/staging/StagingCommitter.java | 7 +-
.../fs/s3a/impl/MultiObjectDeleteSupport.java | 2 +-
.../s3guard/AbstractS3GuardDynamoDBDiagnostic.java | 223 ++++++
.../hadoop/fs/s3a/s3guard/BulkOperationState.java | 6 +
.../fs/s3a/s3guard/DelayedUpdateRenameTracker.java | 12 +-
.../fs/s3a/s3guard/DumpS3GuardDynamoTable.java | 787 +++++++++++++++++++++
.../fs/s3a/s3guard/DynamoDBMetadataStore.java | 428 ++++++++---
.../hadoop/fs/s3a/s3guard/LocalMetadataStore.java | 39 +-
.../hadoop/fs/s3a/s3guard/MetadataStore.java | 44 +-
.../hadoop/fs/s3a/s3guard/NullMetadataStore.java | 16 +-
.../s3guard/PathMetadataDynamoDBTranslation.java | 39 +-
.../fs/s3a/s3guard/PathOrderComparators.java | 4 +-
.../fs/s3a/s3guard/ProgressiveRenameTracker.java | 16 +-
.../fs/s3a/s3guard/PurgeS3GuardDynamoTable.java | 248 +++++++
.../hadoop/fs/s3a/s3guard/RenameTracker.java | 2 +-
.../org/apache/hadoop/fs/s3a/s3guard/S3Guard.java | 48 +-
.../hadoop/fs/s3a/s3guard/S3GuardTableAccess.java | 241 +++++++
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 37 +-
.../site/markdown/tools/hadoop-aws/committers.md | 11 +-
.../src/site/markdown/tools/hadoop-aws/s3guard.md | 22 +-
.../src/site/markdown/tools/hadoop-aws/testing.md | 99 ++-
.../fs/contract/s3a/ITestS3AContractRootDir.java | 18 +-
.../apache/hadoop/fs/s3a/AbstractS3ATestBase.java | 7 +
.../hadoop/fs/s3a/ITestAuthoritativePath.java | 305 ++++++++
.../s3a/ITestS3AMetadataPersistenceException.java | 4 +-
.../hadoop/fs/s3a/ITestS3GuardEmptyDirs.java | 44 ++
.../hadoop/fs/s3a/ITestS3GuardListConsistency.java | 119 ++--
.../fs/s3a/ITestS3GuardOutOfBandOperations.java | 51 +-
.../apache/hadoop/fs/s3a/MockS3AFileSystem.java | 11 +-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 51 ++
.../hadoop/fs/s3a/commit/AbstractCommitITest.java | 89 +--
.../fs/s3a/commit/AbstractITCommitMRJob.java | 13 +-
.../fs/s3a/commit/staging/StagingTestBase.java | 34 +-
.../s3a/commit/staging/TestStagingCommitter.java | 4 +-
.../TestStagingDirectoryOutputCommitter.java | 90 ++-
.../staging/TestStagingPartitionedTaskCommit.java | 38 +-
.../integration/ITestDirectoryCommitProtocol.java | 34 +
.../commit/terasort/AbstractCommitTerasortIT.java | 23 +-
.../fs/s3a/impl/ITestPartialRenamesDeletes.java | 42 --
.../fs/s3a/impl/TestPartialDeleteFailures.java | 18 +-
.../s3a/s3guard/AbstractS3GuardToolTestBase.java | 15 +-
.../fs/s3a/s3guard/ITestDynamoDBMetadataStore.java | 306 +++++---
.../s3guard/ITestDynamoDBMetadataStoreScale.java | 23 +-
.../fs/s3a/s3guard/ITestS3GuardConcurrentOps.java | 4 +-
.../s3a/s3guard/ITestS3GuardDDBRootOperations.java | 269 +++++++
.../fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java | 27 +-
.../fs/s3a/s3guard/MetadataStoreTestBase.java | 95 ++-
.../fs/s3a/s3guard/TestDynamoDBMiscOperations.java | 4 +-
.../scale/AbstractITestS3AMetadataStoreScale.java | 8 +-
.../fs/s3a/scale/ITestLocalMetadataStoreScale.java | 3 +-
.../hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java | 175 +++--
.../hadoop-aws/src/test/resources/log4j.properties | 2 +
.../dynamometer/blockgenerator/BlockInfo.java | 50 +-
.../GenerateDNBlockInfosReducer.java | 2 +-
.../dynamometer/blockgenerator/XMLParser.java | 30 +-
.../dynamometer/blockgenerator/TestXMLParser.java | 31 +-
.../hadoop-dynamometer-infra/pom.xml | 10 +
.../src/main/bash/create-slim-hadoop-tar.sh | 2 +-
.../src/main/bash/upload-fsimage.sh | 4 +-
.../apache/hadoop/tools/dynamometer/AMOptions.java | 9 +-
.../tools/dynamometer/ApplicationMaster.java | 16 +-
.../apache/hadoop/tools/dynamometer/Client.java | 34 +-
.../hadoop/tools/dynamometer/DynoInfraUtils.java | 11 +-
.../tools/dynamometer/SimulatedDataNodes.java | 13 +-
.../src/main/resources/start-component.sh | 9 +-
.../tools/dynamometer/TestDynamometerInfra.java | 169 +++--
.../src/main/bash/parse-start-timestamp.sh | 4 +-
.../workloadgenerator/CreateFileMapper.java | 4 +-
.../workloadgenerator/WorkloadDriver.java | 14 +-
.../workloadgenerator/WorkloadMapper.java | 6 +
.../audit/AuditCommandParser.java | 2 +
.../audit/AuditLogDirectParser.java | 4 +-
.../audit/AuditLogHiveTableParser.java | 4 +-
.../workloadgenerator/audit/AuditReplayMapper.java | 8 +-
.../src/site/markdown/Dynamometer.md | 299 ++++++++
.../src/site/resources/css/site.css | 30 +
.../images/dynamometer-architecture-infra.png | Bin 0 -> 123874 bytes
.../images/dynamometer-architecture-replay.png | Bin 0 -> 159507 bytes
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 8 +
.../distributedshell/TestDistributedShell.java | 66 +-
.../hadoop/yarn/service/client/ServiceClient.java | 2 +-
.../src/main/resources/yarn-default.xml | 17 +-
.../resource-types-error-redefine-fpga-unit.xml | 45 ++
.../resource-types-error-redefine-gpu-unit.xml | 45 ++
.../yarn/server/nodemanager/NodeManager.java | 1 +
.../resources/numa/NumaResourceAllocation.java | 59 +-
.../resources/numa/NumaResourceAllocator.java | 34 +-
.../logaggregation/LogAggregationService.java | 79 ++-
.../resourceplugin/gpu/GpuDiscoverer.java | 62 +-
.../resourceplugin/gpu/GpuResourcePlugin.java | 22 +
.../recovery/NMLeveldbStateStoreService.java | 5 +-
.../webapp/dao/gpu/GpuDeviceInformation.java | 18 +-
.../webapp/dao/gpu/GpuDeviceInformationParser.java | 48 +-
.../webapp/dao/gpu/PerGpuDeviceInformation.java | 1 -
.../webapp/dao/gpu/PerGpuMemoryUsage.java | 7 +-
.../yarn/server/nodemanager/TestNodeManager.java | 28 +
.../logaggregation/TestLogAggregationService.java | 39 +
.../resourceplugin/gpu/TestGpuDiscoverer.java | 19 +-
.../resourceplugin/gpu/TestGpuResourcePlugin.java | 54 ++
.../recovery/TestNMLeveldbStateStoreService.java | 52 +-
.../dao/gpu/TestGpuDeviceInformationParser.java | 157 +++-
.../test/resources/nvidia-smi-output-excerpt.xml | 71 ++
.../resources/nvidia-smi-output-missing-tags.xml | 28 +
.../resources/nvidia-smi-output-missing-tags2.xml | 61 ++
.../test/resources/nvidia-smi-sample-output.xml | 547 ++++++++++++++
.../test/resources/nvidia-smi-sample-xml-output | 547 --------------
.../server/resourcemanager/ResourceManager.java | 1 +
.../server/resourcemanager/rmnode/RMNodeImpl.java | 8 +
.../resourcemanager/scheduler/QueueMetrics.java | 34 +-
.../scheduler/fair/FSLeafQueue.java | 33 +-
.../resourcemanager/TestRMNodeTransitions.java | 18 +
.../yarn/server/resourcemanager/TestRMRestart.java | 5 +-
.../resourcemanager/TestResourceManager.java | 30 +-
.../scheduler/capacity/TestLeafQueue.java | 1 +
.../apache/hadoop/yarn/server/MiniYARNCluster.java | 20 +-
.../pom.xml | 40 +-
.../documentstore/DocumentStoreUtils.java | 27 +-
.../document/entity/TimelineEventSubDoc.java | 7 +-
.../document/entity/TimelineMetricSubDoc.java | 6 -
.../cosmosdb/CosmosDBDocumentStoreReader.java | 115 +--
.../cosmosdb/CosmosDBDocumentStoreWriter.java | 203 ++++--
.../cosmosdb/TestCosmosDBDocumentStoreReader.java | 67 ++
.../MockedCosmosDBDocumentStoreWriter.java | 47 ++
.../cosmosdb/TestCosmosDBDocumentStoreWriter.java | 96 +++
.../src/site/markdown/DockerContainers.md | 86 +++
.../src/site/markdown/FairScheduler.md | 14 +-
.../yarn-queue/fair-queue-conf-table.hbs | 2 +-
.../templates/components/yarn-queue/fair-queue.hbs | 12 +
.../src/main/webapp/app/templates/yarn-queue.hbs | 44 +-
321 files changed, 11633 insertions(+), 3351 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org