You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by si...@apache.org on 2020/04/23 18:00:31 UTC
[hadoop-ozone] branch HDDS-2665-ofs updated (8356a4a -> 9ed2e15)
This is an automated email from the ASF dual-hosted git repository.
siyao pushed a change to branch HDDS-2665-ofs
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from 8356a4a HDDS-3390. Adapt OFSPath to master branch changes (#847)
add eece604 HDDS-3281. Add timeouts to all robot tests (#723)
add a661a75 HDDS-3288: Update default RPC handler SCM/OM count to 100 (#729)
add 099ab62 HDDS-3273. getConf does not return all OM addresses. (#727)
add d7dcb8b Revert "HDDS-3064. Get Key is hung when READ delay is injected in chunk file path. (#673)"
add f5fa408 Revert "HDDS-3234. Fix retry interval default in Ozone client. (#698)"
add 39e116a HDDS-3294. Disable intermittent TestContainerStateMachineFailureOnRead
add ad37f03 HDDS-2011. Ignore TestRandomKeyGenerator.bigFileThan2GB
add b719173 HDDS-3290. Ignore TestRandomKeyGenerator
add d2082b8 HDDS-3297. Ignore TestOzoneClientKeyGenerator.
add a66aae8 HDDS-3290. Disable all the freon integration tests
add f63c5af HDDS-3298. Ignore TestFailureHandlingByClient.testDatanodeExclusionWithMajorityCommit
add a17551e HDDS-3265. Ignore TestRatisPipelineLeader.java
add 557c48e HDDS-3272. Disable hdfs 27 acceptance test
add 386baf1 HDDS-3251. Bump version to 0.6.0-SNAPSHOT
add 79ce00e HDDS-2980. Delete replayed entry from OpenKeyTable during commit (#625)
add 5e23b25 HDDS-3291. Write operation when both OM followers are shutdown. (#733)
add 33ac261 HDDS-2943. Parameterize unit tests for chunk manager implementation (#694)
add 6b5298b HDDS-3160. Disable index and filter block cache for RocksDB
add 94413cd HDDS-3023. Create Freon test to test isolated Ratis LEADER
add 18577a9 HDDS-3236. Fix Dropwizard metrics mapping for latest Ratis metrics
add 2fefd46 HDDS-3142. Create isolated environment for OM to test it without SCM
add 3d3b374 HDDS-3249: remove unrelated assert (#737)
add 97f59f8 HDDS-3248. shutdown defaultMetricsSystem before tests (#704)
add 924b50c HDDS-2621. Enable OM HA acceptance tests. (#265)
add bd1995a HDDS-2184. Rename ozone scmcli to ozone admin (#687)
add f2343f1 HDDS-3156 update allocateContainer to remove additional createPipeline step. (#663)
add 410e4d3 HDDS-2339. Add OzoneManager to MiniOzoneChaosCluster (#643)
add 7faa07f HDDS-3289. Add a freon generator to create nested directories. (#730)
add 32f13f7 HDDS-3289. (Checkstyle addendum) Add a freon generator to create nested directories. (#730)
add f43b33e HDDS-3281. (Addendum: topology) Add timeouts to all robot tests
add ce6ad30 HDDS-2621. Revert/Disable OM HA tests.
add 61b6be0 HDDS-3165. Integrate Recon missing containers UI with endpoint. (#735)
add 1ccf78f HDDS-3047. ObjectStore#listVolumesByUser and CreateVolumeHandler#call should get principal name by default (#582)
add 6384b8d HDDS-3130. Add jaeger trace span in s3gateway
add 8d4aa8d HDDS-3240. Improve write efficiency by creating container in parallel (#734)
add e6c8e4c HDDS-3311. OM logs not available for OM HA acceptance test (#747)
add 6661469 HDDS-3308. Update Ratis snapshot (#741)
add 55d78d6 HDDS-3330. Disable TestDeleteWithSlowFollower test as it has known problem.
add f1da73b HDDS-3324. OM Client fails with StringIndexOutOfBoundsException. (#759)
add 0189f8d HDDS-3334. OM Client failover to next OM on NotLeaderException (#760)
add 63fde40 HDDS-3253. Initialize Recon metrics for prometheus at /prom endpoint
add 1ebadbe HDDS-3173. Provide better default JVM options
add 12fb2f6 HDDS-3316. Checkstyle check fails silently in case of mvn related errors
add 1ceb514 HDDS-3239. Provide message-level metrics from the generic protocol dispatch
add 692cd7a HDDS-3320. Add read only based load generator to MiniOzoneChaosCluster. (#754)
add 9f8ff0e HDDS-3295. Ozone admins getting Permission Denied error while creating volume
add e9742db HDDS-3318. Support /close command in the Github comments
add dc9b24d HDDS-2798. beyond/Containers.md translation
add 598e24c HDDS-3332. Make Robot test invocation independent from Python version (#757)
add b4eafde HDDS-3332. Upgrade Robot tests to Python 3 (#766)
add f04a409 HDDS-3339. Rename CONTRIBUTION.md to CONTRIBUTING.md (#765)
add f2c32e4 HDDS-3337. Export the IPC port of OM in docker-compose
add 065d443 HDDS-3224. Enforce volume and bucket name rule at create time. (#718)
add 7d45380 HDDS-2691. Ozone Shell code cleanup
add 11b0984 HDDS-3341. Checkstyle fails for new modules/versions (#768)
add 876bec0 HDDS-2817. Fix listing buckets for setting --prefix equal to bucket name (#430)
add e54d0b5 HDDS-3237. Recon should provide the list of datanodes that a missing … (#753)
add 0fbacbc HDDS-2793. concept/Datanodes.md translation (#520)
add b80aaad HDDS-3349. Fix two link addresses in README.md (#774)
add d3b81a4 HDDS-3241. Invalid container reported to SCM should be deleted
add 54580ec HDDS-3351. Remove unnecessary dependency Curator. (#777)
add 7164c19 HDDS-3344. Remove Jenkins file (#778)
add 9872253 HDDS-3328. List design docs as part of the documentation page. (#755)
add ad92019 HDDS-3312. Remove sever-side dependencies from hdds/ozone-common
add 15db251 HDDS-3319. Handle HA for BasicOzoneClientAdapterImpl#renew/cancel(). (#752)
add b0973ff HDDS-3280. Ozone BaseHTTPServer should honor ozone.security.enabled c… (#758)
add a1ae540 HDDS-3347. PipelineActionHandler should handle unknown pipeline. (#773)
add de30b37 HDDS-3348. scmcli container info command shows the wrong container state. (#776)
add 89c34b4 HDDS-3353. Remove unnecessary transitive hadoop-common dependencies on server side.
add ccf8bd8 HDDS-3331. Simplify s3bucket -> ozone volume/bucket mapping (#756)
add a24a79b HDDS-3270. Allow safemode listeners to be notified when some precheck rules pass (#719)
add aae0fd3 HDDS-3290. REVERT: Disable all the freon integration tests. (#786)
add e2ebbf8 HDDS-3302. Add ability in MiniOzoneCluster to create multiple storage directories
add b50e932 HDDS-3244. Improve write efficiency by opening RocksDB only once (#709)
add eb177f4 HDDS-3366. Ozone Client should not retry on invalid token (#802)
add ebc6160 HDDS-3340. Move Ozone Shell from ozone-manager to tools
add 1412b6c HDDS-3359. Remove Yetus helper code from ozone
add e31a4ce Revert "HDDS-3291. Write operation when both OM followers are shutdown. (#733)" (#803)
add e71c383 HDDS-3365. Ensure OzoneConfiguration is initialized in OzoneClientFactory#getOzoneClient. (#798)
add 216b41f HDDS-3335. Recon unit tests cleanup. (#772)
add 5d8be37 HDDS-2294. Create a new HISTORY.md in the new repository. (#764)
add 486d749 HDDS-3135. Enable topology acceptance test added in HDDS-3084 to read data when racks stopped (#790)
add c75d4c7 HDDS-3372. Delete HISTORY.txt (#805)
add 90c29d7 HDDS-3093. Allow forced overwrite of local file (#800)
add 129496f HDDS-3368. Ozone filesystem jar should not include webapps folder (#804)
add 09ff517 HDDS-3377. Remove guava 26.0-android jar. (#808)
add b037c93 Handle InterruptedException in OzoneContainer (#801)
add c9a1925 HDDS-2976. Recon throws error while trying to get snapshot over https (#783)
add ac5fb0f HDDS-3374. OMVolumeSetOwnerRequest doesn't check if user is already the owner (#806)
add 25913f7 HDDS-3355. Add wait time between client retries to OM (#784)
add 8287c72 HDDS-3374. Addendum: OMVolumeSetOwnerRequest doesn't check if user is already the owner (#821)
add 506778f HDDS-3134. Debug Tool that gives chunk location information given a key. (#684)
add 5936594 HDDS-2802. tools/AuditParser.md translation (#793)
add 9d0d341 HDDS-3394. Skip generation of encryptionkey for directory create operation. (#823)
add 901a7bc HDDS-3384. Update SpringFramework to 5.1.14. (#829)
add bc8d43e HDDS-3357. Add check for import from shaded package (#788)
add e9a466d HDDS-2682: OM File create request does not check for existing directory with the same name (#824)
add 8712d50 HDDS-3387. Fix ContainerOperationClient#createContainer (#831)
add 1505778 HDDS-3386. Remove unnecessary transitive hadoop-common dependencies on server side (addendum). (#818)
add 17456bc HDDS-3386. Remove unnecessary transitive hadoop-common dependencies on server side (addendum) -- fix checkstyle
add 3699e79 HDDS-3291. Write operation when both OM followers are shutdown. (#815)
add f020e6a HDDS-3392.OM create key/file should not generate different data encryption key during validateAndUpdateCache. (#830)
add 67ca42c HDDS-3401. Ozone audit entries could be consistent among volume creation with quota and update quota (#832)
add 2f37a25 HDDS-3322. StandAlone Pipelines are created in an infinite loop (#749)
add 57c9cc9 HDDS-3360. Duplicate Ozone Client in renew/cancel token handlers (#799)
add 151fe08 HDDS-2575. Handle InterruptedException in LogSubcommand
add f58cae9 HDDS-3336. Metrics for Recon OzoneManager DB sync.
add 8b4d3f5 HDDS-3406. Remove RetryInvocation INFO logging from ozone CLI output (#838)
add 77bb315 HDDS-3400. Extract test utilities to separate module
add 09fd46b HDDS-3403. Generate ozone specific version from type in FSProto.proto
add 94beb2a HDDS-2911. lastUsed and stateEnterTime value in container info is not human friendly
add 3bb5838 Update grpc/netty dependencies (#840)
add 8c80e3c Revert "Update grpc/netty dependencies (#840)" (#841)
add e800d2b HDDS-3397. Make jmh jar dependencies optional
add 491d69a HDDS-3456. Fix Acceptance test failures due to disk out of space. (#844)
add ce94889 HDDS-3380. MiniOzoneHAClusterImpl#initOMRatisConf will reset the conf… (#817)
add ffb340e HDDS-3101. Depend on lightweight ConfigurationSource interface instead of Hadoop Configuration
add fc4d951 HDDS-3468. Organize log4j dependency in pom.xml. (#849)
add 2acd2b1 HDDS-3399. Update JaegerTracing
add 6afe2a8 HDDS-3471. Update to latest Ratis Snapshot 0.6.0-490b689-SNAPSHOT. (#851)
add fac26e1 HDDS-3139. Pipeline placement should max out pipeline usage (#668)
add a59ff25 HDDS-3451. Add timeout to XceiverServerRatis#submitRequest call. (#845)
add 091993b HDDS-3056. Allow users to list volumes they have access to, and optionally allow all users to list all volumes (#696)
add 1b7f617 HDDS-3388. Add bucket encryption key info to bucket create audit log. (#820)
add ca6da46 HDDS-2809. recipe/_index.md translation (#795)
add 00799c2 HDDS-3393. Recon throws NPE in clusterState endpoint (#854)
add 36cdf5a HDDS-3467. OM Client RPC failover retries happening more than configured. (#853)
add 706fe1b HDDS-3168. Improve read efficiency by merging a lot of RPC call getContainerWithPipeline into one. (#692)
add 2098516 HDDS-3155. Improved ozone client flush implementation to make it faster. (#716)
add 0466ade HDDS-3172. Use DBStore instead of MetadataStore in SCM
add fa3aaf1 HDDS-2810. recipe/Prometheus.md translation (#796)
add 811e18c HDDS-3373. Intermittent failure in TestDnRatisLogParser and TestOMRatisLogParser (#858)
add 52f68c1 HDDS-2801. tools/Genconf.md translation (#792)
add cadf414 HDDS-2804. tools/TestTools.md (#794)
add b1b12a1 HDDS-3472. Remove redundant hdds.version from hadoop-hdds/common/pom.xml (#852)
add d980e5a HDDS-3483. Handle unhealthy replica state of an open container.
add 266a9e8 HDDS-3464. Use dedicated build partition for acceptance tests in github actions environment
new dd99223 Merge remote-tracking branch 'asf/master' into HDDS-2665-ofs
new 9ed2e15 HDDS-3390. Rebase OFS branch - 2. Adapt OFS classes to HDDS-3101 (#822)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/comment-commands/close.sh | 41 ++
.github/comment-commands/pending.sh | 7 +
.github/comment-commands/ready.sh | 6 +
.github/workflows/post-commit.yml | 68 +-
.github/workflows/pr.yml | 68 +-
CONTRIBUTION.md => CONTRIBUTING.md | 0
HISTORY.md | 61 ++
README.md | 6 +-
dev-support/bin/qbt | 18 -
dev-support/bin/smart-apply-patch | 18 -
dev-support/bin/test-patch | 18 -
dev-support/bin/yetus-wrapper | 188 -----
dev-support/byteman/mock-scm.btm | 34 +
dev-support/byteman/ratis-mock-followers.btm | 42 ++
hadoop-hdds/client/pom.xml | 13 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 84 ++-
.../hadoop/hdds/scm/XceiverClientManager.java | 10 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 57 +-
.../hadoop/hdds/scm/client/HddsClientUtils.java | 49 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 15 +-
hadoop-hdds/common/pom.xml | 61 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 6 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 25 +-
.../hadoop/hdds/annotation/InterfaceStability.java | 3 -
.../org/apache/hadoop/hdds/cli/GenericCli.java | 12 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 193 +----
.../hadoop/hdds/freon/FakeClusterTopology.java | 92 +++
.../freon/FakeScmBlockLocationProtocolClient.java | 100 +++
.../FakeScmContainerLocationProtocolClient.java | 76 ++
.../apache/hadoop/hdds/freon}/package-info.java | 14 +-
.../java/org/apache/hadoop/hdds/fs/DUFactory.java | 11 +-
.../hdds/fs/DedicatedDiskSpaceUsageFactory.java | 11 +-
.../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 26 +-
.../hdds/function/SupplierWithIOException.java | 19 +-
.../hdds/ratis/ContainerCommandRequestMessage.java | 7 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 32 +-
.../hadoop/hdds/scm/ByteStringConversion.java | 4 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 24 +
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 4 +-
.../hadoop/hdds/scm/container/ContainerInfo.java | 56 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 4 +-
.../hadoop/hdds/scm/net/NodeSchemaManager.java | 4 +-
.../protocol/StorageContainerLocationProtocol.java | 12 +
...inerLocationProtocolClientSideTranslatorPB.java | 42 +-
.../hadoop/hdds/security/x509/SecurityConfig.java | 22 +-
.../hadoop/hdds/tracing/GrpcServerInterceptor.java | 11 +-
.../apache/hadoop/hdds/tracing/TraceAllMethod.java | 8 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 71 +-
.../utils/LegacyHadoopConfigurationSource.java | 74 ++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 30 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +-
.../org/apache/hadoop/ozone/OzoneSecurityUtil.java | 30 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
.../org/apache/hadoop/ozone/lock/LockManager.java | 6 +-
.../java/org/apache/hadoop/ozone/package-info.java | 21 +-
.../proto/StorageContainerLocationProtocol.proto | 12 +
.../common/src/main/resources/ozone-default.xml | 82 ++-
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 7 +-
.../hadoop/hdds/conf/TestOzoneConfiguration.java | 18 +-
.../org/apache/hadoop/hdds/fs/TestDUFactory.java | 8 +-
.../fs/TestDedicatedDiskSpaceUsageFactory.java | 8 +-
.../hadoop/hdds/fs/TestSpaceUsageFactory.java | 40 +-
.../hdds/scm/net/TestNetworkTopologyImpl.java | 38 +-
.../hadoop/hdds/scm/net/TestNodeSchemaManager.java | 16 +-
hadoop-hdds/config/pom.xml | 13 +-
.../hdds/conf/ConfigurationReflectionUtil.java | 159 ++++
.../hadoop/hdds/conf/ConfigurationSource.java | 291 ++++++++
.../org/apache/hadoop/hdds/conf/StorageSize.java | 102 +++
.../org/apache/hadoop/hdds/conf/StorageUnit.java | 529 +++++++++++++
.../apache/hadoop/hdds/conf/TimeDurationUtil.java | 154 ++++
hadoop-hdds/container-service/pom.xml | 17 +-
.../hadoop/hdds/freon/FakeRatisFollower.java | 125 ++++
.../apache/hadoop/hdds/freon}/package-info.java | 8 +-
.../hadoop/ozone/HddsDatanodeHttpServer.java | 4 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 4 +-
.../container/common/helpers/ContainerMetrics.java | 4 +-
.../container/common/impl/ChunkLayOutVersion.java | 62 +-
.../ozone/container/common/impl/ContainerData.java | 19 +
.../container/common/impl/HddsDispatcher.java | 87 ++-
.../ozone/container/common/interfaces/Handler.java | 8 +-
.../container/common/report/ReportManager.java | 21 +-
.../container/common/report/ReportPublisher.java | 27 +-
.../common/report/ReportPublisherFactory.java | 32 +-
.../common/statemachine/DatanodeStateMachine.java | 51 +-
.../common/statemachine/EndpointStateMachine.java | 6 +-
.../common/statemachine/SCMConnectionManager.java | 63 +-
.../common/statemachine/StateContext.java | 62 +-
.../CreatePipelineCommandHandler.java | 29 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 6 +-
.../ReplicateContainerCommandHandler.java | 6 +-
.../common/states/datanode/InitDatanodeState.java | 32 +-
.../states/datanode/RunningDatanodeState.java | 6 +-
.../states/endpoint/HeartbeatEndpointTask.java | 10 +-
.../states/endpoint/RegisterEndpointTask.java | 10 +-
.../states/endpoint/VersionEndpointTask.java | 23 +-
.../common/transport/server/XceiverServerGrpc.java | 39 +-
.../server/ratis/ContainerStateMachine.java | 110 ++-
.../transport/server/ratis/XceiverServerRatis.java | 118 +--
.../container/common/utils/ContainerCache.java | 39 +-
.../ozone/container/common/volume/HddsVolume.java | 25 +-
.../container/common/volume/HddsVolumeChecker.java | 4 +-
.../container/common/volume/MutableVolumeSet.java | 13 +-
.../ozone/container/common/volume/VolumeInfo.java | 17 +-
.../container/keyvalue/KeyValueContainer.java | 49 +-
.../container/keyvalue/KeyValueContainerCheck.java | 133 ++--
.../container/keyvalue/KeyValueContainerData.java | 24 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 115 +--
.../container/keyvalue/helpers/BlockUtils.java | 36 +-
.../container/keyvalue/helpers/ChunkUtils.java | 39 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 23 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 6 +-
.../keyvalue/impl/ChunkManagerFactory.java | 4 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 23 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 25 +-
.../keyvalue/interfaces/ChunkManager.java | 1 -
.../background/BlockDeletingService.java | 81 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 30 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 9 +-
.../replication/ContainerStreamingOutput.java | 44 --
.../replication/SimpleContainerDownloader.java | 4 +-
...inerDatanodeProtocolServerSideTranslatorPB.java | 3 +-
.../ozone/container/common/ContainerTestUtils.java | 24 +-
.../ozone/container/common/SCMTestUtils.java | 23 +-
.../container/common/TestBlockDeletingService.java | 80 +-
.../ozone/container/common/TestContainerCache.java | 2 -
.../container/common/TestDatanodeStateMachine.java | 60 +-
.../common/TestKeyValueContainerData.java | 18 +-
.../common/impl/TestContainerDataYaml.java | 21 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 21 +-
.../common/impl/TestContainerPersistence.java | 98 +--
.../container/common/impl/TestContainerSet.java | 23 +-
.../container/common/impl/TestHddsDispatcher.java | 23 +-
.../container/common/interfaces/TestHandler.java | 19 +-
.../container/common/report/TestReportManager.java | 11 +-
.../common/report/TestReportPublisher.java | 4 +-
.../common/report/TestReportPublisherFactory.java | 14 +-
.../TestCloseContainerCommandHandler.java | 21 +-
.../states/endpoint/TestHeartbeatEndpointTask.java | 52 +-
.../container/common/volume/TestHddsVolume.java | 28 +-
.../common/volume/TestHddsVolumeChecker.java | 40 +-
.../volume/TestRoundRobinVolumeChoosingPolicy.java | 22 +-
.../common/volume/TestVolumeSetDiskChecks.java | 16 +-
.../container/keyvalue/ChunkLayoutTestInfo.java | 120 +++
.../keyvalue/TestKeyValueBlockIterator.java | 58 +-
.../container/keyvalue/TestKeyValueContainer.java | 23 +-
.../keyvalue/TestKeyValueContainerCheck.java | 76 +-
.../TestKeyValueContainerMarkUnhealthy.java | 16 +-
.../container/keyvalue/TestKeyValueHandler.java | 164 +++--
.../container/keyvalue/TestTarContainerPacker.java | 18 +-
.../keyvalue/impl/AbstractTestChunkManager.java | 170 +++++
.../keyvalue/impl/CommonChunkManagerTestCases.java | 201 +++++
.../keyvalue/impl/TestBlockManagerImpl.java | 17 +-
.../container/keyvalue/impl/TestChunkManager.java | 325 --------
.../keyvalue/impl/TestChunkManagerDummyImpl.java | 60 ++
.../keyvalue/impl/TestFilePerBlockStrategy.java | 140 ++++
.../keyvalue/impl/TestFilePerChunkStrategy.java | 92 +++
.../container/ozoneimpl/TestOzoneContainer.java | 19 +-
.../replication/TestReplicationSupervisor.java | 32 +-
.../testutils/BlockDeletingServiceTestImpl.java | 16 +-
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 10 +-
hadoop-hdds/docs/archetypes/design.md | 60 ++
hadoop-hdds/docs/content/beyond/Containers.zh.md | 203 +++++
hadoop-hdds/docs/content/concept/Datanodes.md | 4 +-
hadoop-hdds/docs/content/concept/Datanodes.zh.md | 30 +-
hadoop-hdds/docs/content/design/configless.md | 31 +
hadoop-hdds/docs/content/design/decommissioning.md | 17 +-
hadoop-hdds/docs/content/design/gdpr.md | 31 +
hadoop-hdds/docs/content/design/locks.md | 29 +
hadoop-hdds/docs/content/design/multiraft.md | 31 +
.../docs/content/design/namespace-support.md | 29 +
hadoop-hdds/docs/content/design/nfs.md | 31 +
hadoop-hdds/docs/content/design/ofs.md | 33 +
hadoop-hdds/docs/content/design/omha.md | 33 +
.../content/design/ozone-enhancement-proposals.md | 2 +-
.../docs/content/design/ozone-volume-management.md | 181 +++++
hadoop-hdds/docs/content/design/recon1.md | 32 +
hadoop-hdds/docs/content/design/recon2.md | 29 +
hadoop-hdds/docs/content/design/s3gateway.md | 31 +
hadoop-hdds/docs/content/design/scmha.md | 29 +
hadoop-hdds/docs/content/design/tde.md | 29 +
hadoop-hdds/docs/content/design/token.md | 25 +
hadoop-hdds/docs/content/design/trash.md | 25 +
hadoop-hdds/docs/content/design/typesafeconfig.md | 33 +
hadoop-hdds/docs/content/recipe/Prometheus.zh.md | 93 +++
.../{tools/SCMCLI.md => recipe/_index.zh.md} | 15 +-
hadoop-hdds/docs/content/shell/VolumeCommands.md | 6 +-
.../docs/content/shell/VolumeCommands.zh.md | 5 +-
hadoop-hdds/docs/content/tools/Admin.md | 35 +
hadoop-hdds/docs/content/tools/AuditParser.zh.md | 68 ++
.../Datanodes.zh.md => tools/Genconf.zh.md} | 11 +-
hadoop-hdds/docs/content/tools/TestTools.md | 8 +-
.../tools/{TestTools.md => TestTools.zh.md} | 53 +-
hadoop-hdds/docs/content/tools/_index.md | 4 +-
hadoop-hdds/docs/pom.xml | 4 +-
.../ozonedoc/layouts/_default/baseof.html} | 31 +-
.../themes/ozonedoc/layouts/design/section.html | 68 ++
.../themes/ozonedoc/layouts/design/single.html | 47 ++
.../themes/ozonedoc/layouts/partials/sidebar.html | 1 +
hadoop-hdds/framework/pom.xml | 49 +-
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 0
.../hdds/conf/DatanodeRatisServerConfig.java | 0
.../apache/hadoop/hdds/conf/HddsConfServlet.java | 30 +-
.../org/apache/hadoop/hdds/conf/package-info.java | 0
.../certificates/utils/CertificateSignRequest.java | 23 +-
.../certificates/utils/SelfSignedCertificate.java | 27 +-
.../hdds/security/x509/keys/HDDSKeyGenerator.java | 13 +-
.../org/apache/hadoop/hdds/server}/JsonUtils.java | 8 +-
.../server/OzoneProtocolMessageDispatcher.java | 18 +-
.../org/apache/hadoop/hdds/server/ServerUtils.java | 12 +-
.../hadoop/hdds/server/http/BaseHttpServer.java | 36 +-
.../hadoop/hdds/server/http/FilterInitializer.java | 4 +-
.../apache/hadoop/hdds/server/http/HttpConfig.java | 4 +-
.../hadoop/hdds/server/http/HttpServer2.java | 98 +--
.../server/http/RatisNameRewriteSampleBuilder.java | 17 +-
.../hdds/server/http/StaticUserWebFilter.java | 6 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 39 +-
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 13 +-
.../hadoop/hdds/utils/ProtocolMessageMetrics.java | 42 +-
.../hdds/utils/db/BatchOperationHandler.java | 38 +-
.../hdds/utils/db/DBColumnFamilyDefinition.java | 81 ++
.../apache/hadoop/hdds/utils/db/DBDefinition.java} | 39 +-
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 1 -
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 18 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 56 +-
.../apache/hadoop/hdds/utils/db/RDBMetrics.java | 13 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 2 +-
.../apache/hadoop/hdds/server/TestJsonUtils.java | 37 +-
.../apache/hadoop/hdds/server/TestServerUtils.java | 21 +-
.../hdds/server/http/TestBaseHttpServer.java | 4 +-
.../hdds/server/http/TestRatisNameRewrite.java | 10 +
.../hadoop/hdds/utils/TestMetadataStore.java | 13 +-
.../hadoop/hdds/utils/TestRocksDBStoreMBean.java | 23 +-
hadoop-hdds/hadoop-dependency-client/README.md | 60 ++
hadoop-hdds/hadoop-dependency-client/pom.xml | 282 +++++++
.../{tools => hadoop-dependency-server}/pom.xml | 49 +-
.../{client => hadoop-dependency-test}/pom.xml | 26 +-
hadoop-hdds/pom.xml | 34 +-
hadoop-hdds/server-scm/pom.xml | 26 +-
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 35 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 33 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 34 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 41 +-
.../hdds/scm/container/ContainerManager.java | 6 +-
.../hdds/scm/container/ContainerReportHandler.java | 57 +-
.../hdds/scm/container/ContainerStateManager.java | 61 +-
.../hdds/scm/container/ReplicationManager.java | 9 +-
.../hdds/scm/container/SCMContainerManager.java | 145 ++--
.../ContainerPlacementPolicyFactory.java | 11 +-
.../algorithms/SCMContainerPlacementCapacity.java | 4 +-
.../algorithms/SCMContainerPlacementRackAware.java | 4 +-
.../algorithms/SCMContainerPlacementRandom.java | 4 +-
.../container/placement/metrics/ContainerStat.java | 2 +-
.../hadoop/hdds/scm/metadata/ContainerIDCodec.java | 48 ++
.../hdds/scm/metadata/ContainerInfoCodec.java | 47 ++
.../hadoop/hdds/scm/metadata/PipelineCodec.java | 56 ++
.../hadoop/hdds/scm/metadata/PipelineIDCodec.java | 45 ++
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 98 +++
.../hadoop/hdds/scm/metadata/SCMMetadataStore.java | 27 +-
.../hdds/scm/metadata/SCMMetadataStoreRDBImpl.java | 113 +--
.../hadoop/hdds/scm/node/NewNodeHandler.java | 15 +-
.../hadoop/hdds/scm/node/NodeStateManager.java | 55 +-
.../scm/node/NonHealthyToHealthyNodeHandler.java | 4 +-
.../hadoop/hdds/scm/node/StaleNodeHandler.java | 4 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 15 +-
.../hdds/scm/pipeline/PipelineActionHandler.java | 63 +-
.../hadoop/hdds/scm/pipeline/PipelineFactory.java | 6 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 140 ++--
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 75 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 76 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 11 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 178 +++--
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 12 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 15 +-
...lockLocationProtocolServerSideTranslatorPB.java | 4 +-
...inerLocationProtocolServerSideTranslatorPB.java | 27 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 4 +-
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 4 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 11 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 4 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 60 +-
.../hadoop/hdds/scm/safemode/SafeModeHandler.java | 23 +-
.../hadoop/hdds/scm/safemode/SafeModePrecheck.java | 4 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 6 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 185 +++--
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 21 +-
.../hdds/scm/server/StorageContainerManager.java | 88 ++-
.../server/StorageContainerManagerHttpServer.java | 8 +-
.../apache/hadoop/hdds/scm/TestHddsServerUtil.java | 18 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 42 +-
.../scm/TestStorageContainerManagerHttpServer.java | 6 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 37 +-
.../container/TestCloseContainerEventHandler.java | 38 +-
.../scm/container/TestContainerReportHandler.java | 4 +-
.../TestIncrementalContainerReportHandler.java | 4 +-
.../hdds/scm/container/TestReplicationManager.java | 34 +-
.../scm/container/TestSCMContainerManager.java | 65 +-
.../scm/container/TestUnknownContainerReport.java | 145 ++++
.../algorithms/TestContainerPlacementFactory.java | 19 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 4 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../hdds/scm/node/TestContainerPlacement.java | 65 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 14 +-
.../scm/pipeline/MockRatisPipelineProvider.java | 20 +-
.../scm/pipeline/TestPipelineActionHandler.java | 69 ++
.../scm/pipeline/TestPipelinePlacementPolicy.java | 93 ++-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 178 ++++-
.../scm/pipeline/TestSimplePipelineProvider.java | 2 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 38 +-
.../TestOneReplicaPipelineSafeModeRule.java | 19 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 179 ++++-
.../hdds/scm/safemode/TestSafeModeHandler.java | 71 +-
.../ozone/container/common/TestEndPoint.java | 7 +-
.../placement/TestContainerPlacement.java | 32 +-
.../upgrade => hadoop-hdds/test-utils}/pom.xml | 50 +-
.../org/apache/hadoop/test/GenericTestUtils.java | 365 +++++++++
.../org/apache/hadoop/test/LambdaTestUtils.java | 814 +++++++++++++++++++++
.../apache/hadoop/test/TimedOutTestsListener.java | 183 +++++
.../java/org/apache/hadoop/test}/package-info.java | 7 +-
hadoop-hdds/tools/pom.xml | 13 +-
.../hdds/scm/cli/ContainerOperationClient.java | 24 +-
.../hdds/scm/cli/ReplicationManagerCommands.java | 21 +-
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 114 ---
.../hadoop/hdds/scm/cli/SafeModeCommands.java | 16 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 26 +-
.../hdds/scm/cli/container/CloseSubcommand.java | 4 +-
.../hdds/scm/cli/container/ContainerCommands.java | 32 +-
.../hdds/scm/cli/container/DeleteSubcommand.java | 3 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 20 +-
.../hdds/scm/cli/container/ListSubcommand.java | 21 +-
.../hdds/scm/cli/container/WithScmClient.java | 13 +-
.../hdds/scm/cli/datanode/DatanodeCommands.java | 22 +-
.../hdds/scm/cli/pipeline/PipelineCommands.java | 22 +-
hadoop-ozone/Jenkinsfile | 116 ---
hadoop-ozone/client/pom.xml | 9 +-
.../apache/hadoop/ozone/client/ObjectStore.java | 6 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 8 +-
.../apache/hadoop/ozone/client/OzoneClient.java | 4 +-
.../hadoop/ozone/client/OzoneClientFactory.java | 69 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 18 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 36 +-
.../client/io/BlockOutputStreamEntryPool.java | 8 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 16 +-
.../hadoop/ozone/client/rpc/OzoneKMSUtil.java | 32 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 91 ++-
.../hadoop/ozone/client/TestHddsClientUtils.java | 41 +-
hadoop-ozone/common/pom.xml | 29 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 166 +++--
.../apache/hadoop/ozone/conf/OMClientConfig.java | 67 ++
.../apache/hadoop/ozone/conf}/package-info.java | 11 +-
.../apache/hadoop/ozone/freon/OzoneGetConf.java | 27 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 5 +-
.../hadoop/ozone/om/exceptions/OMException.java | 2 +
.../ozone/om/exceptions/OMReplayException.java | 16 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 105 ++-
.../apache/hadoop/ozone/om/helpers/DBUpdates.java | 58 ++
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 2 +
.../hadoop/ozone/om/helpers/OzoneFileStatus.java | 116 ++-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 24 +-
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 4 +-
.../ozone/om/protocol/OzoneManagerProtocol.java | 20 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 127 ++--
.../ozone/security/OzoneTokenIdentifier.java | 22 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 7 +-
.../common/src/main/proto/FSProtos.proto | 2 +-
.../src/main/proto/OzoneManagerProtocol.proto | 6 +
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 125 +---
hadoop-ozone/csi/pom.xml | 4 +-
hadoop-ozone/datanode/pom.xml | 6 +-
hadoop-ozone/dev-support/checks/author.sh | 6 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 14 +-
.../intellij/runConfigurations/OzoneShell.xml | 6 +-
hadoop-ozone/dist/pom.xml | 16 +-
.../main/compose/ozone-hdfs/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-mr/common-config | 2 +-
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 1 +
.../ozone-mr/hadoop27/{test.sh => run-test.sh} | 0
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 1 +
.../src/main/compose/ozone-mr/hadoop31/test.sh | 6 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-om-ha/.ssh/id_rsa | 15 -
.../src/main/compose/ozone-om-ha/.ssh/id_rsa.pub | 15 -
.../src/main/compose/ozone-om-ha/docker-config | 3 +-
.../ozone-om-ha/{run.sh => test_disabled.sh} | 0
.../dist/src/main/compose/ozone-topology/.env | 2 +-
.../compose/ozone-topology/docker-compose.yaml | 3 +-
.../src/main/compose/ozone-topology/docker-config | 2 +-
.../src/main/compose/ozone-topology/hdds-3084.sh | 56 --
.../dist/src/main/compose/ozone-topology/test.sh | 20 +-
.../src/main/compose/ozone/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
.../main/compose/ozoneblockade/docker-compose.yaml | 1 +
.../compose/ozones3-haproxy/docker-compose.yaml | 1 +
.../main/compose/ozonescripts/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 12 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 1 +
.../src/main/compose/ozonesecure-mr/docker-config | 2 +-
.../main/compose/ozonesecure/docker-compose.yaml | 1 +
.../src/main/compose/ozonesecure/docker-config | 1 +
.../dist/src/main/compose/ozonesecure/test.sh | 2 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 2 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 24 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 8 -
.../licenses/LICENSE-org.openjdk.jmh-jmh-core.txt | 347 ---------
...SE-org.openjdk.jmh-jmh-generator-annprocess.txt | 347 ---------
.../smoketest/{scmcli => admincli}/datanode.robot | 3 +-
.../smoketest/{scmcli => admincli}/pipeline.robot | 5 +-
.../main/smoketest/auditparser/auditparser.robot | 1 +
.../dist/src/main/smoketest/basic/basic.robot | 1 +
.../src/main/smoketest/basic/ozone-shell.robot | 10 +-
.../dist/src/main/smoketest/createbucketenv.robot | 2 +-
.../dist/src/main/smoketest/createmrenv.robot | 4 +-
.../dist/src/main/smoketest/env-compose.robot | 2 +-
.../dist/src/main/smoketest/freon/freon.robot | 1 +
.../dist/src/main/smoketest/gdpr/gdpr.robot | 1 +
.../dist/src/main/smoketest/mapreduce.robot | 4 +-
.../main/smoketest/om-ratis/testOMAdminCmd.robot | 2 +-
.../dist/src/main/smoketest/omha/testOMHA.robot | 45 +-
.../src/main/smoketest/ozonefs/hadoopo3fs.robot | 5 +-
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 5 +
.../dist/src/main/smoketest/recon/recon-api.robot | 4 +
.../src/main/smoketest/s3/MultipartUpload.robot | 1 +
.../dist/src/main/smoketest/s3/awss3.robot | 1 +
.../dist/src/main/smoketest/s3/bucketcreate.robot | 1 +
.../dist/src/main/smoketest/s3/buckethead.robot | 1 +
.../dist/src/main/smoketest/s3/bucketlist.robot | 1 +
.../dist/src/main/smoketest/s3/objectcopy.robot | 1 +
.../dist/src/main/smoketest/s3/objectdelete.robot | 1 +
.../src/main/smoketest/s3/objectmultidelete.robot | 1 +
.../dist/src/main/smoketest/s3/objectputget.robot | 1 +
.../dist/src/main/smoketest/s3/webui.robot | 1 +
.../main/smoketest/security/ozone-secure-fs.robot | 1 +
.../main/smoketest/security/ozone-secure-s3.robot | 1 +
.../smoketest/security/ozone-secure-token.robot | 76 ++
.../smoketest/topology/{scmcli.robot => cli.robot} | 5 +-
.../src/main/smoketest/topology/loaddata.robot | 1 +
.../src/main/smoketest/topology/readdata.robot | 1 +
.../src/shell/conf/ozone-shell-log4j.properties | 1 +
.../dist/src/shell/hdds/hadoop-functions.sh | 63 ++
hadoop-ozone/dist/src/shell/ozone/ozone | 68 +-
hadoop-ozone/dist/src/test/shell/gc_opts.bats | 44 ++
.../fault-injection-test/mini-chaos-tests/pom.xml | 14 +-
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 252 +++++--
.../ozone/MiniOzoneDatanodeChaosCluster.java | 57 ++
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 14 +-
.../hadoop/ozone/MiniOzoneOMChaosCluster.java | 132 ++++
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 46 +-
.../ozone/loadgenerators/AgedLoadGenerator.java | 20 +-
.../hadoop/ozone/loadgenerators/DataBuffer.java | 1 +
.../loadgenerators/FilesystemLoadGenerator.java | 15 +-
.../hadoop/ozone/loadgenerators/LoadExecutors.java | 25 +-
.../hadoop/ozone/loadgenerators/LoadGenerator.java | 13 +-
.../ozone/loadgenerators/RandomLoadGenerator.java | 10 +-
...adGenerator.java => ReadOnlyLoadGenerator.java} | 49 +-
.../org/apache/hadoop/ozone/utils/LoadBucket.java | 17 +-
.../fault-injection-test/network-tests/pom.xml | 2 +-
hadoop-ozone/fault-injection-test/pom.xml | 4 +-
hadoop-ozone/insight/pom.xml | 30 +-
.../apache/hadoop/ozone/insight/LogSubcommand.java | 2 +
hadoop-ozone/integration-test/pom.xml | 16 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 112 ++-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 7 -
.../ozone/contract/ITestOzoneContractCreate.java | 5 +-
.../ozone/contract/ITestOzoneContractDelete.java | 5 +-
.../ozone/contract/ITestOzoneContractDistCp.java | 5 +-
.../contract/ITestOzoneContractGetFileStatus.java | 5 +-
.../fs/ozone/contract/ITestOzoneContractMkdir.java | 5 +-
.../fs/ozone/contract/ITestOzoneContractOpen.java | 5 +-
.../ozone/contract/ITestOzoneContractRename.java | 5 +-
.../ozone/contract/ITestOzoneContractRootDir.java | 6 +-
.../fs/ozone/contract/ITestOzoneContractSeek.java | 5 +-
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 6 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 33 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 12 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 157 +++-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 3 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 45 +-
.../ozone/TestStorageContainerManagerHelper.java | 10 +-
.../ozone/client/CertificateClientTestImpl.java | 24 +-
.../ozone/client/rpc/Test2BlockOutputStream.java | 173 +++++
.../TestContainerStateMachineFailureOnRead.java | 16 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 24 +-
.../client/rpc/TestFailureHandlingByClient.java | 23 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 23 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 23 +-
.../ozone/dn/ratis/TestDnRatisLogParser.java | 21 +-
.../ozone/freon/TestOzoneClientKeyGenerator.java | 13 +-
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 8 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 37 +-
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 116 ++-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 27 +-
.../ozone/om/TestOzoneManagerListVolumes.java | 238 ++++++
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 2 +-
.../ozone/om/parser/TestOMRatisLogParser.java | 5 +-
.../{ozShell => shell}/TestOzoneDatanodeShell.java | 2 +-
.../ozone/{ozShell => shell}/TestOzoneShellHA.java | 6 +-
.../ozone/{ozShell => shell}/TestS3Shell.java | 20 +-
hadoop-ozone/ozone-manager/pom.xml | 17 +-
.../apache/hadoop/ozone/om/KeyDeletingService.java | 4 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 70 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 91 ++-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 0
.../java/org/apache/hadoop/ozone/om/OMStorage.java | 4 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 2 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 173 +++--
.../hadoop/ozone/om/OzoneManagerHttpServer.java | 10 +-
.../hadoop/ozone/om/S3SecretManagerImpl.java | 0
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 22 +-
.../hadoop/ozone/om/codec/OmBucketInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 0
.../ozone/om/codec/OmMultipartKeyInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmPrefixInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmVolumeArgsCodec.java | 0
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 0
.../hadoop/ozone/om/codec/S3SecretValueCodec.java | 0
.../ozone/om/codec/TokenIdentifierCodec.java | 0
.../hadoop/ozone/om/codec/UserVolumeInfoCodec.java | 0
.../apache/hadoop/ozone/om/codec/package-info.java | 0
.../ozone/om/ratis/OzoneManagerRatisServer.java | 37 +-
.../om/request/bucket/OMBucketCreateRequest.java | 4 +
.../om/request/file/OMDirectoryCreateRequest.java | 42 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 13 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 60 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 7 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 80 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 13 +-
.../S3InitiateMultipartUploadRequest.java | 6 -
.../S3MultipartUploadCommitPartRequest.java | 69 +-
.../S3MultipartUploadCompleteRequest.java | 78 +-
.../om/request/volume/OMVolumeCreateRequest.java | 20 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 21 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 26 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 2 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 43 +-
.../S3MultipartUploadCommitPartResponse.java | 59 +-
.../S3MultipartUploadCompleteResponse.java | 48 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 21 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 6 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 13 +-
.../protocolPB/OzoneManagerRequestHandler.java | 26 +-
.../hadoop/ozone/security/AWSV4AuthValidator.java | 0
.../security/OzoneBlockTokenSecretManager.java | 0
.../OzoneDelegationTokenSecretManager.java | 99 ++-
.../hadoop/ozone/security/OzoneSecretKey.java | 0
.../hadoop/ozone/security/OzoneSecretManager.java | 0
.../hadoop/ozone/security/OzoneSecretStore.java | 0
.../hadoop/ozone/security}/package-info.java | 7 +-
.../web/ozShell/bucket/AddAclBucketHandler.java | 105 ---
.../web/ozShell/bucket/CreateBucketHandler.java | 107 ---
.../web/ozShell/bucket/DeleteBucketHandler.java | 64 --
.../web/ozShell/bucket/GetAclBucketHandler.java | 85 ---
.../web/ozShell/bucket/InfoBucketHandler.java | 68 --
.../web/ozShell/bucket/ListBucketHandler.java | 105 ---
.../web/ozShell/bucket/RemoveAclBucketHandler.java | 106 ---
.../web/ozShell/bucket/SetAclBucketHandler.java | 100 ---
.../ozone/web/ozShell/keys/AddAclKeyHandler.java | 109 ---
.../ozone/web/ozShell/keys/DeleteKeyHandler.java | 70 --
.../ozone/web/ozShell/keys/GetAclKeyHandler.java | 89 ---
.../ozone/web/ozShell/keys/GetKeyHandler.java | 113 ---
.../ozone/web/ozShell/keys/InfoKeyHandler.java | 78 --
.../ozone/web/ozShell/keys/ListKeyHandler.java | 113 ---
.../ozone/web/ozShell/keys/PutKeyHandler.java | 129 ----
.../web/ozShell/keys/RemoveAclKeyHandler.java | 109 ---
.../ozone/web/ozShell/keys/SetAclKeyHandler.java | 102 ---
.../ozone/web/ozShell/s3/GetS3SecretHandler.java | 63 --
.../web/ozShell/token/CancelTokenHandler.java | 76 --
.../ozone/web/ozShell/token/GetTokenHandler.java | 81 --
.../ozone/web/ozShell/token/PrintTokenHandler.java | 71 --
.../ozone/web/ozShell/token/RenewTokenHandler.java | 79 --
.../web/ozShell/volume/AddAclVolumeHandler.java | 103 ---
.../web/ozShell/volume/CreateVolumeHandler.java | 103 ---
.../web/ozShell/volume/DeleteVolumeHandler.java | 62 --
.../web/ozShell/volume/GetAclVolumeHandler.java | 80 --
.../web/ozShell/volume/ListVolumeHandler.java | 111 ---
.../web/ozShell/volume/RemoveAclVolumeHandler.java | 103 ---
.../web/ozShell/volume/SetAclVolumeHandler.java | 106 ---
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 13 +-
.../org/apache/hadoop/ozone/om/TestOMStorage.java | 9 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 26 +-
.../ozone/om/TestOzoneManagerHttpServer.java | 21 +-
.../om/codec/TestOmMultipartKeyInfoCodec.java | 0
.../ozone/om/codec/TestOmPrefixInfoCodec.java | 0
.../ozone/om/codec/TestS3SecretValueCodec.java | 0
.../apache/hadoop/ozone/om/codec/package-info.java | 0
.../request/bucket/TestOMBucketCreateRequest.java | 5 +
.../om/request/key/TestOMKeyCommitRequest.java | 42 ++
.../s3/bucket/TestS3BucketCreateRequest.java | 4 +-
.../request/volume/TestOMVolumeCreateRequest.java | 5 +
.../volume/TestOMVolumeSetOwnerRequest.java | 49 ++
.../om/response/key/TestOMKeyCommitResponse.java | 14 +-
.../ozone/security/TestAWSV4AuthValidator.java | 0
.../TestOzoneDelegationTokenSecretManager.java | 14 +-
.../ozone/security/TestOzoneTokenIdentifier.java | 41 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 17 +-
hadoop-ozone/ozonefs-lib-legacy/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 36 +-
.../java/org/apache/hadoop/fs/ozone/BasicOzFs.java | 12 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 13 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 20 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 9 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 18 +-
.../org/apache/hadoop/fs/ozone/O3fsDtFetcher.java | 8 +-
.../main/java/org/apache/hadoop/fs/ozone/OzFs.java | 12 +-
.../hadoop/fs/ozone/OzoneClientAdapterImpl.java | 5 +-
.../apache/hadoop/fs/ozone/OzoneFileSystem.java | 10 +-
.../org/apache/hadoop/fs/ozone/OzoneFsShell.java | 5 +-
.../fs/ozone/RootedOzoneClientAdapterImpl.java | 6 +-
.../hadoop/fs/ozone/RootedOzoneFileSystem.java | 16 +-
.../fs/ozone/TestOzoneFileSystemWithMocks.java | 31 +-
.../hadoop/fs/ozone/TestReadWriteStatistics.java | 41 +-
.../ozone/TestRootedOzoneFileSystemWithMocks.java | 5 +-
hadoop-ozone/pom.xml | 29 +-
hadoop-ozone/recon-codegen/pom.xml | 7 +-
.../ozone/recon/codegen/JooqCodeGenerator.java | 2 +-
.../recon/codegen/ReconSchemaGenerationModule.java | 2 +
.../recon/schema/ContainerSchemaDefinition.java | 87 +++
.../recon/schema/ReconTaskSchemaDefinition.java | 2 +
.../ozone/recon/schema/StatsSchemaDefinition.java | 2 +
.../recon/schema/UtilizationSchemaDefinition.java | 19 +-
hadoop-ozone/recon/pom.xml | 33 +-
.../hadoop/ozone/recon/ConfigurationProvider.java | 5 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 51 +-
.../hadoop/ozone/recon/ReconSchemaManager.java | 4 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 10 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 54 +-
.../ozone/recon/api/ClusterStateEndpoint.java | 36 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 29 +-
.../hadoop/ozone/recon/api/types/KeysResponse.java | 61 +-
.../recon/api/types/MissingContainerMetadata.java | 14 +-
.../ozone/recon/fsck/MissingContainerTask.java | 22 +-
.../recon/metrics/OzoneManagerSyncMetrics.java | 133 ++++
.../hadoop/ozone/recon/metrics}/package-info.java | 16 +-
.../recon/persistence/ContainerSchemaManager.java | 104 +++
.../recon/recovery/ReconOMMetadataManager.java | 6 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 10 +
.../ozone/recon/scm/ReconContainerManager.java | 55 +-
.../hadoop/ozone/recon/scm/ReconDBDefinition.java} | 31 +-
.../recon/scm/ReconDatanodeProtocolServer.java | 4 +-
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 17 +-
.../ozone/recon/scm/ReconPipelineFactory.java | 4 +-
.../ozone/recon/scm/ReconPipelineManager.java | 34 +-
.../recon/scm/ReconPipelineReportHandler.java | 4 +-
.../scm/ReconStorageContainerManagerFacade.java | 46 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 10 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 76 +-
.../webapps/recon/ozone-recon-web/api/db.json | 74 +-
.../webapps/recon/ozone-recon-web/api/routes.json | 3 +-
.../recon/ozone-recon-web/src/utils/common.tsx | 9 +-
.../views/MissingContainers/MissingContainers.less | 4 +
.../views/MissingContainers/MissingContainers.tsx | 99 ++-
.../src/views/Overview/Overview.tsx | 12 +-
...erTest.java => OMMetadataManagerTestUtils.java} | 60 +-
.../hadoop/ozone/recon/ReconTestInjector.java | 337 +++++++++
.../apache/hadoop/ozone/recon/TestReconUtils.java | 45 +-
.../ozone/recon/api/TestContainerEndpoint.java | 170 ++---
.../hadoop/ozone/recon/api/TestEndpoints.java | 141 ++--
.../ozone/recon/api/TestTaskStatusService.java | 23 +-
.../ozone/recon/fsck/TestMissingContainerTask.java | 40 +-
...tabaseTest.java => AbstractReconSqlDBTest.java} | 131 +++-
.../TestReconInternalSchemaDefinition.java | 26 +-
.../persistence/TestStatsSchemaDefinition.java | 25 +-
.../TestUtilizationSchemaDefinition.java | 37 +-
.../scm/AbstractReconContainerManagerTest.java | 38 +-
.../ozone/recon/scm/TestReconContainerManager.java | 2 +-
.../ozone/recon/scm/TestReconPipelineManager.java | 37 +-
.../recon/scm/TestReconPipelineReportHandler.java | 2 +-
.../impl/TestContainerDBServiceProviderImpl.java | 56 +-
.../impl/TestOzoneManagerServiceProviderImpl.java | 64 +-
.../recon/tasks/TestContainerKeyMapperTask.java | 71 +-
.../ozone/recon/tasks/TestFileSizeCountTask.java | 16 +-
.../recon/tasks/TestReconTaskControllerImpl.java | 28 +-
.../recon/types/GuiceInjectorUtilsForTests.java | 136 ----
hadoop-ozone/s3gateway/pom.xml | 36 +-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 2 +
.../hadoop/ozone/s3/S3GatewayHttpServer.java | 4 +-
.../org/apache/hadoop/ozone/s3/TracingFilter.java | 82 +++
hadoop-ozone/tools/pom.xml | 31 +-
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 58 +-
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 12 +-
.../hadoop/ozone/debug/ChunkDataNodeDetails.java} | 31 +-
.../apache/hadoop/ozone/debug/ChunkDetails.java} | 44 +-
.../apache/hadoop/ozone/debug/ChunkKeyHandler.java | 171 +++++
.../hadoop/ozone/debug/ContainerChunkInfo.java | 93 +++
.../org/apache/hadoop/ozone/debug/OzoneDebug.java | 49 +-
.../apache/hadoop/ozone/debug}/package-info.java | 7 +-
.../hadoop/ozone/freon/BaseAppendLogGenerator.java | 10 +
.../hadoop/ozone/freon/BaseFreonGenerator.java | 17 +-
.../freon/FollowerAppendLogEntryGenerator.java | 8 -
.../java/org/apache/hadoop/ozone/freon/Freon.java | 4 +-
.../apache/hadoop/ozone/freon/FreonHttpServer.java | 4 +-
.../ozone/freon/HadoopNestedDirGenerator.java | 113 +++
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 273 +++++++
.../hadoop/ozone/freon/RandomKeyGenerator.java | 27 +-
.../ozone/genesis/BenchMarkContainerStateMap.java | 8 +-
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 5 +-
.../ozone/genesis/BenchMarkOzoneManager.java | 11 +-
.../apache/hadoop/ozone/genesis/BenchMarkSCM.java | 11 +-
.../ozone/genesis/BenchmarkChunkManager.java | 33 +-
.../apache/hadoop/ozone/genesis/GenesisUtil.java | 58 +-
.../org/apache/hadoop/ozone/shell/Handler.java | 133 ++++
.../org/apache/hadoop/ozone/shell/ListOptions.java | 58 ++
.../apache/hadoop/ozone/shell}/OzoneAddress.java | 45 +-
.../org/apache/hadoop/ozone/shell}/OzoneShell.java | 22 +-
.../java/org/apache/hadoop/ozone/shell}/Shell.java | 27 +-
.../hadoop/ozone/shell/StoreTypeOption.java} | 36 +-
.../apache/hadoop/ozone/shell/acl/AclHandler.java | 62 ++
.../apache/hadoop/ozone/shell/acl/AclOption.java | 91 +++
.../hadoop/ozone/shell/acl/GetAclHandler.java} | 24 +-
.../hadoop/ozone/shell/acl}/package-info.java | 4 +-
.../ozone/shell/bucket/AddAclBucketHandler.java} | 36 +-
.../hadoop/ozone/shell}/bucket/BucketCommands.java | 4 +-
.../hadoop/ozone/shell/bucket/BucketHandler.java} | 23 +-
.../hadoop/ozone/shell/bucket/BucketUri.java | 44 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 88 +++
.../ozone/shell/bucket/DeleteBucketHandler.java} | 33 +-
.../ozone/shell/bucket/GetAclBucketHandler.java} | 28 +-
.../ozone/shell/bucket/InfoBucketHandler.java} | 32 +-
.../ozone/shell/bucket/ListBucketHandler.java | 68 ++
.../shell/bucket/RemoveAclBucketHandler.java} | 36 +-
.../ozone/shell/bucket/SetAclBucketHandler.java} | 37 +-
.../hadoop/ozone/shell/bucket}/package-info.java | 4 +-
.../hadoop/ozone/shell/keys/AddAclKeyHandler.java} | 36 +-
.../hadoop/ozone/shell/keys/DeleteKeyHandler.java | 50 ++
.../hadoop/ozone/shell/keys/GetAclKeyHandler.java} | 28 +-
.../hadoop/ozone/shell/keys/GetKeyHandler.java | 98 +++
.../hadoop/ozone/shell/keys/InfoKeyHandler.java | 58 ++
.../hadoop/ozone/shell}/keys/KeyCommands.java | 4 +-
.../hadoop/ozone/shell/keys/KeyHandler.java} | 23 +-
.../org/apache/hadoop/ozone/shell/keys/KeyUri.java | 44 +-
.../hadoop/ozone/shell/keys/ListKeyHandler.java | 78 ++
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 108 +++
.../ozone/shell/keys/RemoveAclKeyHandler.java} | 36 +-
.../hadoop/ozone/shell}/keys/RenameKeyHandler.java | 45 +-
.../hadoop/ozone/shell/keys/SetAclKeyHandler.java} | 37 +-
.../hadoop/ozone/shell/keys}/package-info.java | 4 +-
.../apache/hadoop/ozone/shell}/package-info.java | 7 +-
.../hadoop/ozone/shell/s3/GetS3SecretHandler.java} | 30 +-
.../hadoop/ozone/shell}/s3/S3BucketMapping.java | 42 +-
.../apache/hadoop/ozone/shell}/s3/S3Handler.java | 30 +-
.../org/apache/hadoop/ozone/shell}/s3/S3Shell.java | 17 +-
.../hadoop/ozone/shell}/s3/package-info.java | 2 +-
.../ozone/shell/token/CancelTokenHandler.java} | 29 +-
.../hadoop/ozone/shell/token/GetTokenHandler.java | 72 ++
.../ozone/shell/token/PrintTokenHandler.java} | 46 +-
.../ozone/shell/token/RenewTokenHandler.java} | 29 +-
.../hadoop/ozone/shell/token/RenewerOption.java} | 32 +-
.../hadoop/ozone/shell}/token/TokenCommands.java | 5 +-
.../hadoop/ozone/shell/token/TokenHandler.java | 54 ++
.../hadoop/ozone/shell/token/TokenOption.java | 56 ++
.../hadoop/ozone/shell}/token/package-info.java | 6 +-
.../ozone/shell/volume/AddAclVolumeHandler.java} | 36 +-
.../ozone/shell/volume/CreateVolumeHandler.java | 74 ++
.../ozone/shell/volume/DeleteVolumeHandler.java} | 30 +-
.../ozone/shell/volume/GetAclVolumeHandler.java} | 28 +-
.../ozone/shell}/volume/InfoVolumeHandler.java | 36 +-
.../ozone/shell/volume/ListVolumeHandler.java | 99 +++
.../shell/volume/RemoveAclVolumeHandler.java} | 36 +-
.../ozone/shell/volume/SetAclVolumeHandler.java} | 37 +-
.../ozone/shell}/volume/UpdateVolumeHandler.java | 46 +-
.../hadoop/ozone/shell}/volume/VolumeCommands.java | 4 +-
.../hadoop/ozone/shell/volume/VolumeHandler.java} | 23 +-
.../hadoop/ozone/shell/volume/VolumeUri.java | 44 +-
.../hadoop/ozone/shell/volume}/package-info.java | 4 +-
.../hadoop/ozone/shell}/TestOzoneAddress.java | 2 +-
.../apache/hadoop/ozone/shell}/package-info.java | 2 +-
hadoop-ozone/upgrade/pom.xml | 9 +-
pom.xml | 47 +-
768 files changed, 19435 insertions(+), 11107 deletions(-)
create mode 100755 .github/comment-commands/close.sh
rename CONTRIBUTION.md => CONTRIBUTING.md (100%)
create mode 100644 HISTORY.md
delete mode 100755 dev-support/bin/qbt
delete mode 100755 dev-support/bin/smart-apply-patch
delete mode 100755 dev-support/bin/test-patch
delete mode 100755 dev-support/bin/yetus-wrapper
create mode 100644 dev-support/byteman/mock-scm.btm
create mode 100644 dev-support/byteman/ratis-mock-followers.btm
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/freon/FakeClusterTopology.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/freon/FakeScmBlockLocationProtocolClient.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/freon/FakeScmContainerLocationProtocolClient.java
rename {hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/freon}/package-info.java (78%)
copy hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/SupplierWithIOException.java (71%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/LegacyHadoopConfigurationSource.java
create mode 100644 hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationReflectionUtil.java
create mode 100644 hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationSource.java
create mode 100644 hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/StorageSize.java
create mode 100644 hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/StorageUnit.java
create mode 100644 hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/TimeDurationUtil.java
create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/freon/FakeRatisFollower.java
copy {hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/codec => hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/freon}/package-info.java (78%)
delete mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ContainerStreamingOutput.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/ChunkLayoutTestInfo.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/AbstractTestChunkManager.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/CommonChunkManagerTestCases.java
delete mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestChunkManager.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestChunkManagerDummyImpl.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestFilePerBlockStrategy.java
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestFilePerChunkStrategy.java
create mode 100644 hadoop-hdds/docs/archetypes/design.md
create mode 100644 hadoop-hdds/docs/content/beyond/Containers.zh.md
create mode 100644 hadoop-hdds/docs/content/design/configless.md
create mode 100644 hadoop-hdds/docs/content/design/gdpr.md
create mode 100644 hadoop-hdds/docs/content/design/locks.md
create mode 100644 hadoop-hdds/docs/content/design/multiraft.md
create mode 100644 hadoop-hdds/docs/content/design/namespace-support.md
create mode 100644 hadoop-hdds/docs/content/design/nfs.md
create mode 100644 hadoop-hdds/docs/content/design/ofs.md
create mode 100644 hadoop-hdds/docs/content/design/omha.md
create mode 100644 hadoop-hdds/docs/content/design/ozone-volume-management.md
create mode 100644 hadoop-hdds/docs/content/design/recon1.md
create mode 100644 hadoop-hdds/docs/content/design/recon2.md
create mode 100644 hadoop-hdds/docs/content/design/s3gateway.md
create mode 100644 hadoop-hdds/docs/content/design/scmha.md
create mode 100644 hadoop-hdds/docs/content/design/tde.md
create mode 100644 hadoop-hdds/docs/content/design/token.md
create mode 100644 hadoop-hdds/docs/content/design/trash.md
create mode 100644 hadoop-hdds/docs/content/design/typesafeconfig.md
create mode 100644 hadoop-hdds/docs/content/recipe/Prometheus.zh.md
rename hadoop-hdds/docs/content/{tools/SCMCLI.md => recipe/_index.zh.md} (55%)
create mode 100644 hadoop-hdds/docs/content/tools/Admin.md
create mode 100644 hadoop-hdds/docs/content/tools/AuditParser.zh.md
copy hadoop-hdds/docs/content/{concept/Datanodes.zh.md => tools/Genconf.zh.md} (77%)
copy hadoop-hdds/docs/content/tools/{TestTools.md => TestTools.zh.md} (69%)
copy hadoop-hdds/docs/{content/concept/Datanodes.zh.md => themes/ozonedoc/layouts/_default/baseof.html} (64%)
create mode 100644 hadoop-hdds/docs/themes/ozonedoc/layouts/design/section.html
create mode 100644 hadoop-hdds/docs/themes/ozonedoc/layouts/design/single.html
rename hadoop-hdds/{common => framework}/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisGrpcConfig.java (100%)
rename hadoop-hdds/{common => framework}/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java (100%)
rename hadoop-hdds/{common => framework}/src/main/java/org/apache/hadoop/hdds/conf/HddsConfServlet.java (93%)
copy hadoop-hdds/{common => framework}/src/main/java/org/apache/hadoop/hdds/conf/package-info.java (100%)
rename hadoop-hdds/{common/src/main/java/org/apache/hadoop/ozone/web/utils => framework/src/main/java/org/apache/hadoop/hdds/server}/JsonUtils.java (98%)
copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/recovery/ReconOMMetadataManager.java => hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/BatchOperationHandler.java (50%)
create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBColumnFamilyDefinition.java
copy hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineProvider.java => framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBDefinition.java} (51%)
rename hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/web/ozShell/TestObjectPrinter.java => hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/TestJsonUtils.java (53%)
create mode 100644 hadoop-hdds/hadoop-dependency-client/README.md
create mode 100644 hadoop-hdds/hadoop-dependency-client/pom.xml
copy hadoop-hdds/{tools => hadoop-dependency-server}/pom.xml (60%)
copy hadoop-hdds/{client => hadoop-dependency-test}/pom.xml (75%)
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/ContainerIDCodec.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/ContainerInfoCodec.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineCodec.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineIDCodec.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/SCMDBDefinition.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestUnknownContainerReport.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineActionHandler.java
copy {hadoop-ozone/upgrade => hadoop-hdds/test-utils}/pom.xml (56%)
create mode 100644 hadoop-hdds/test-utils/src/main/java/org/apache/hadoop/test/GenericTestUtils.java
create mode 100644 hadoop-hdds/test-utils/src/main/java/org/apache/hadoop/test/LambdaTestUtils.java
create mode 100644 hadoop-hdds/test-utils/src/main/java/org/apache/hadoop/test/TimedOutTestsListener.java
copy hadoop-hdds/{common/src/main/java/org/apache/hadoop/ozone/web/utils => test-utils/src/main/java/org/apache/hadoop/test}/package-info.java (92%)
delete mode 100644 hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SCMCLI.java
copy hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/WithScmClient.java (76%)
delete mode 100644 hadoop-ozone/Jenkinsfile
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/conf/OMClientConfig.java
rename {hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/web/utils => hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/conf}/package-info.java (79%)
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/DBUpdates.java
rename {hadoop-hdds => hadoop-ozone}/common/src/main/proto/FSProtos.proto (97%)
rename hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop27/{test.sh => run-test.sh} (100%)
rename hadoop-ozone/dist/src/main/compose/ozone-om-ha/{run.sh => test_disabled.sh} (100%)
delete mode 100755 hadoop-ozone/dist/src/main/compose/ozone-topology/hdds-3084.sh
delete mode 100644 hadoop-ozone/dist/src/main/license/bin/licenses/LICENSE-org.openjdk.jmh-jmh-core.txt
delete mode 100644 hadoop-ozone/dist/src/main/license/bin/licenses/LICENSE-org.openjdk.jmh-jmh-generator-annprocess.txt
rename hadoop-ozone/dist/src/main/smoketest/{scmcli => admincli}/datanode.robot (91%)
rename hadoop-ozone/dist/src/main/smoketest/{scmcli => admincli}/pipeline.robot (86%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-token.robot
rename hadoop-ozone/dist/src/main/smoketest/topology/{scmcli.robot => cli.robot} (88%)
create mode 100644 hadoop-ozone/dist/src/test/shell/gc_opts.bats
create mode 100644 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneDatanodeChaosCluster.java
create mode 100644 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneOMChaosCluster.java
copy hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/{FilesystemLoadGenerator.java => ReadOnlyLoadGenerator.java} (55%)
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2BlockOutputStream.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerListVolumes.java
rename hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/{ozShell => shell}/TestOzoneDatanodeShell.java (99%)
rename hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/{ozShell => shell}/TestOzoneShellHA.java (98%)
rename hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/{ozShell => shell}/TestS3Shell.java (93%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/S3SecretManagerImpl.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/OmBucketInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/OmKeyInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/OmMultipartKeyInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/OmPrefixInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/OmVolumeArgsCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/RepeatedOmKeyInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/S3SecretValueCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/TokenIdentifierCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/UserVolumeInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/codec/package-info.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/AWSV4AuthValidator.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java (88%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretKey.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretManager.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretStore.java (100%)
copy hadoop-ozone/ozone-manager/src/{test/java/org/apache/hadoop/ozone/web/ozShell => main/java/org/apache/hadoop/ozone/security}/package-info.java (91%)
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/AddAclBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/CreateBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/DeleteBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/GetAclBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/InfoBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/ListBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/RemoveAclBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket/SetAclBucketHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/AddAclKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/DeleteKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/GetAclKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/GetKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/InfoKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/ListKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/PutKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/RemoveAclKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/SetAclKeyHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/s3/GetS3SecretHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/token/CancelTokenHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/token/GetTokenHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/token/PrintTokenHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/token/RenewTokenHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/AddAclVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/CreateVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/DeleteVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/GetAclVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/ListVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/RemoveAclVolumeHandler.java
delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume/SetAclVolumeHandler.java
rename hadoop-ozone/{common => ozone-manager}/src/test/java/org/apache/hadoop/ozone/om/codec/TestOmMultipartKeyInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/test/java/org/apache/hadoop/ozone/om/codec/TestOmPrefixInfoCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/test/java/org/apache/hadoop/ozone/om/codec/TestS3SecretValueCodec.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/test/java/org/apache/hadoop/ozone/om/codec/package-info.java (100%)
rename hadoop-ozone/{common => ozone-manager}/src/test/java/org/apache/hadoop/ozone/security/TestAWSV4AuthValidator.java (100%)
create mode 100644 hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ContainerSchemaDefinition.java
create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
rename {hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/conf => hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics}/package-info.java (81%)
create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/ContainerSchemaManager.java
copy hadoop-ozone/{fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/LoadGenerator.java => recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconDBDefinition.java} (56%)
rename hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/{AbstractOMMetadataManagerTest.java => OMMetadataManagerTestUtils.java} (79%)
create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/ReconTestInjector.java
rename hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/{AbstractSqlDatabaseTest.java => AbstractReconSqlDBTest.java} (50%)
delete mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/types/GuiceInjectorUtilsForTests.java
create mode 100644 hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/TracingFilter.java
copy hadoop-ozone/{recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => tools/src/main/java/org/apache/hadoop/ozone/debug/ChunkDataNodeDetails.java} (59%)
copy hadoop-ozone/{common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMReplayException.java => tools/src/main/java/org/apache/hadoop/ozone/debug/ChunkDetails.java} (55%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/ChunkKeyHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/ContainerChunkInfo.java
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DatanodeCommands.java => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/OzoneDebug.java (52%)
copy hadoop-ozone/{ozone-manager/src/test/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/debug}/package-info.java (91%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/LeaderAppendLogEntryGenerator.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/Handler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/ListOptions.java
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/OzoneAddress.java (89%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/OzoneShell.java (77%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/Shell.java (67%)
copy hadoop-ozone/{common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMReplayException.java => tools/src/main/java/org/apache/hadoop/ozone/shell/StoreTypeOption.java} (54%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/acl/AclHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/acl/AclOption.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/acl/GetAclHandler.java} (64%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys => tools/src/main/java/org/apache/hadoop/ozone/shell/acl}/package-info.java (88%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/AddAclBucketHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/bucket/BucketCommands.java (95%)
copy hadoop-ozone/{recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketHandler.java} (66%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DatanodeCommands.java => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketUri.java (50%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/CreateBucketHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/DeleteBucketHandler.java} (55%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/GetAclBucketHandler.java} (60%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/InfoBucketHandler.java} (56%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/ListBucketHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/RemoveAclBucketHandler.java} (50%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/SetAclBucketHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/volume => tools/src/main/java/org/apache/hadoop/ozone/shell/bucket}/package-info.java (88%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/keys/AddAclKeyHandler.java} (51%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/DeleteKeyHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/keys/GetAclKeyHandler.java} (61%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/GetKeyHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/InfoKeyHandler.java
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/keys/KeyCommands.java (95%)
copy hadoop-ozone/{recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => tools/src/main/java/org/apache/hadoop/ozone/shell/keys/KeyHandler.java} (67%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DatanodeCommands.java => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/KeyUri.java (50%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/PutKeyHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/keys/RemoveAclKeyHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/keys/RenameKeyHandler.java (55%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/keys/SetAclKeyHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/bucket => tools/src/main/java/org/apache/hadoop/ozone/shell/keys}/package-info.java (88%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/package-info.java (86%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/s3/GetS3SecretHandler.java} (54%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/s3/S3BucketMapping.java (58%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/s3/S3Handler.java (64%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/s3/S3Shell.java (83%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/s3/package-info.java (94%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/token/CancelTokenHandler.java} (56%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/token/GetTokenHandler.java
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/Handler.java => tools/src/main/java/org/apache/hadoop/ozone/shell/token/PrintTokenHandler.java} (52%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/token/RenewTokenHandler.java} (55%)
copy hadoop-ozone/{common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMReplayException.java => tools/src/main/java/org/apache/hadoop/ozone/shell/token/RenewerOption.java} (58%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/token/TokenCommands.java (95%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/token/TokenHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/token/TokenOption.java
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/token/package-info.java (86%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/AddAclVolumeHandler.java} (50%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/CreateVolumeHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/DeleteVolumeHandler.java} (57%)
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/GetAclVolumeHandler.java} (60%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/volume/InfoVolumeHandler.java (54%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/ListVolumeHandler.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/RemoveAclVolumeHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/ObjectPrinter.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/SetAclVolumeHandler.java} (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/volume/UpdateVolumeHandler.java (55%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell => tools/src/main/java/org/apache/hadoop/ozone/shell}/volume/VolumeCommands.java (95%)
rename hadoop-ozone/{recon/src/test/java/org/apache/hadoop/ozone/recon/GuiceInjectorUtilsForTestsImpl.java => tools/src/main/java/org/apache/hadoop/ozone/shell/volume/VolumeHandler.java} (66%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DatanodeCommands.java => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/VolumeUri.java (50%)
rename hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys => tools/src/main/java/org/apache/hadoop/ozone/shell/volume}/package-info.java (88%)
rename hadoop-ozone/{ozone-manager/src/test/java/org/apache/hadoop/ozone/web/ozShell => tools/src/test/java/org/apache/hadoop/ozone/shell}/TestOzoneAddress.java (98%)
rename hadoop-ozone/{ozone-manager/src/test/java/org/apache/hadoop/ozone/web/ozShell => tools/src/test/java/org/apache/hadoop/ozone/shell}/package-info.java (94%)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 02/02: HDDS-3390. Rebase OFS branch - 2. Adapt OFS
classes to HDDS-3101 (#822)
Posted by si...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
siyao pushed a commit to branch HDDS-2665-ofs
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 9ed2e15834c8d19c2dac30c2d06788006e31ec82
Author: Siyao Meng <sm...@cloudera.com>
AuthorDate: Wed Apr 22 23:19:53 2020 -0700
HDDS-3390. Rebase OFS branch - 2. Adapt OFS classes to HDDS-3101 (#822)
---
.../fs/ozone/BasicRootedOzoneClientAdapterImpl.java | 9 ++++++---
.../hadoop/fs/ozone/BasicRootedOzoneFileSystem.java | 18 ++++++++++++++----
.../hadoop/fs/ozone/RootedOzoneClientAdapterImpl.java | 6 ++++--
.../apache/hadoop/fs/ozone/RootedOzoneFileSystem.java | 16 ++++++++--------
.../fs/ozone/TestRootedOzoneFileSystemWithMocks.java | 5 ++---
5 files changed, 34 insertions(+), 20 deletions(-)
diff --git a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java
index 4409891..bf7b124 100644
--- a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java
+++ b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.hdds.client.ReplicationFactor;
import org.apache.hadoop.hdds.client.ReplicationType;
+import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.security.x509.SecurityConfig;
@@ -123,7 +124,7 @@ public class BasicRootedOzoneClientAdapterImpl
}
public BasicRootedOzoneClientAdapterImpl(String omHost, int omPort,
- Configuration hadoopConf) throws IOException {
+ ConfigurationSource hadoopConf) throws IOException {
ClassLoader contextClassLoader =
Thread.currentThread().getContextClassLoader();
@@ -706,7 +707,8 @@ public class BasicRootedOzoneClientAdapterImpl
Token<OzoneTokenIdentifier> ozoneDt =
(Token<OzoneTokenIdentifier>) token;
OzoneClient ozoneClient =
- OzoneClientFactory.getRpcClient(conf);
+ OzoneClientFactory.getOzoneClient(OzoneConfiguration.of(conf),
+ ozoneDt);
return ozoneClient.getObjectStore().renewDelegationToken(ozoneDt);
}
@@ -716,7 +718,8 @@ public class BasicRootedOzoneClientAdapterImpl
Token<OzoneTokenIdentifier> ozoneDt =
(Token<OzoneTokenIdentifier>) token;
OzoneClient ozoneClient =
- OzoneClientFactory.getRpcClient(conf);
+ OzoneClientFactory.getOzoneClient(OzoneConfiguration.of(conf),
+ ozoneDt);
ozoneClient.getObjectStore().cancelDelegationToken(ozoneDt);
}
}
diff --git a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
index ff84175..39aeef1 100644
--- a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
+++ b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
@@ -33,6 +33,9 @@ import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdds.conf.ConfigurationSource;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.LegacyHadoopConfigurationSource;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.om.exceptions.OMException;
import org.apache.hadoop.security.UserGroupInformation;
@@ -136,9 +139,16 @@ public class BasicRootedOzoneFileSystem extends FileSystem {
boolean isolatedClassloader =
conf.getBoolean("ozone.fs.isolated-classloader", defaultValue);
- // adapter should be initialized in operations.
- this.adapter = createAdapter(
- conf, omHostOrServiceId, omPort, isolatedClassloader);
+ ConfigurationSource source;
+ if (conf instanceof OzoneConfiguration) {
+ source = (ConfigurationSource) conf;
+ } else {
+ source = new LegacyHadoopConfigurationSource(conf);
+ }
+ this.adapter =
+ createAdapter(source,
+ omHostOrServiceId, omPort,
+ isolatedClassloader);
try {
this.userName =
@@ -155,7 +165,7 @@ public class BasicRootedOzoneFileSystem extends FileSystem {
}
}
- protected OzoneClientAdapter createAdapter(Configuration conf,
+ protected OzoneClientAdapter createAdapter(ConfigurationSource conf,
String omHost, int omPort, boolean isolatedClassloader)
throws IOException {
diff --git a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneClientAdapterImpl.java b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneClientAdapterImpl.java
index e8ac316..e5cfb14 100644
--- a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneClientAdapterImpl.java
+++ b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneClientAdapterImpl.java
@@ -18,7 +18,8 @@
package org.apache.hadoop.fs.ozone;
import java.io.IOException;
-import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
/**
@@ -44,7 +45,8 @@ public class RootedOzoneClientAdapterImpl
}
public RootedOzoneClientAdapterImpl(String omHost, int omPort,
- Configuration hadoopConf, OzoneFSStorageStatistics storageStatistics)
+ ConfigurationSource hadoopConf,
+ OzoneFSStorageStatistics storageStatistics)
throws IOException {
super(omHost, omPort, hadoopConf);
this.storageStatistics = storageStatistics;
diff --git a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneFileSystem.java b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneFileSystem.java
index a8bd099..474e6b5 100644
--- a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneFileSystem.java
+++ b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/RootedOzoneFileSystem.java
@@ -18,9 +18,9 @@
package org.apache.hadoop.fs.ozone;
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.hdds.annotation.InterfaceStability;
+import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.crypto.key.KeyProvider;
import org.apache.hadoop.crypto.key.KeyProviderTokenIssuer;
import org.apache.hadoop.fs.FileSystem;
@@ -32,12 +32,12 @@ import java.io.IOException;
import java.net.URI;
/**
- * The Ozone Filesystem implementation.
+ * The Rooted Ozone Filesystem implementation.
* <p>
* This subclass is marked as private as code should not be creating it
- * directly; use {@link FileSystem#get(Configuration)} and variants to create
- * one. If cast to {@link RootedOzoneFileSystem}, extra methods and features
- * may be accessed. Consider those private and unstable.
+ * directly; use {@link FileSystem#get(org.apache.hadoop.conf.Configuration)}
+ * and variants to create one. If cast to {@link OzoneFileSystem}, extra
+ * methods and features may be accessed. Consider those private and unstable.
*/
@InterfaceAudience.Private
@InterfaceStability.Evolving
@@ -84,7 +84,7 @@ public class RootedOzoneFileSystem extends BasicRootedOzoneFileSystem
}
@Override
- protected OzoneClientAdapter createAdapter(Configuration conf,
+ protected OzoneClientAdapter createAdapter(ConfigurationSource conf,
String omHost, int omPort, boolean isolatedClassloader)
throws IOException {
diff --git a/hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemWithMocks.java b/hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemWithMocks.java
index e278d0c..5ea20a3 100644
--- a/hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemWithMocks.java
+++ b/hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystemWithMocks.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.fs.ozone;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.OmUtils;
@@ -49,7 +48,7 @@ public class TestRootedOzoneFileSystemWithMocks {
@Test
public void testFSUriWithHostPortOverrides() throws Exception {
- Configuration conf = new OzoneConfiguration();
+ OzoneConfiguration conf = new OzoneConfiguration();
OzoneClient ozoneClient = mock(OzoneClient.class);
ObjectStore objectStore = mock(ObjectStore.class);
@@ -80,7 +79,7 @@ public class TestRootedOzoneFileSystemWithMocks {
@Test
public void testFSUriWithHostPortUnspecified() throws Exception {
- Configuration conf = new OzoneConfiguration();
+ OzoneConfiguration conf = new OzoneConfiguration();
final int omPort = OmUtils.getOmRpcPort(conf);
OzoneClient ozoneClient = mock(OzoneClient.class);
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 01/02: Merge remote-tracking branch 'asf/master'
into HDDS-2665-ofs
Posted by si...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
siyao pushed a commit to branch HDDS-2665-ofs
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit dd992234b9d3a585dd6bc6119436075a80b2ca8f
Merge: 8356a4a 266a9e8
Author: Siyao Meng <sm...@cloudera.com>
AuthorDate: Thu Apr 23 10:57:47 2020 -0700
Merge remote-tracking branch 'asf/master' into HDDS-2665-ofs
Conflicts:
hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
.github/comment-commands/close.sh | 41 ++
.github/comment-commands/pending.sh | 7 +
.github/comment-commands/ready.sh | 6 +
.github/workflows/post-commit.yml | 68 +-
.github/workflows/pr.yml | 68 +-
CONTRIBUTION.md => CONTRIBUTING.md | 0
HISTORY.md | 61 ++
README.md | 6 +-
dev-support/bin/qbt | 18 -
dev-support/bin/smart-apply-patch | 18 -
dev-support/bin/test-patch | 18 -
dev-support/bin/yetus-wrapper | 188 -----
dev-support/byteman/mock-scm.btm | 34 +
dev-support/byteman/ratis-mock-followers.btm | 42 ++
hadoop-hdds/client/pom.xml | 13 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 84 ++-
.../hadoop/hdds/scm/XceiverClientManager.java | 10 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 57 +-
.../hadoop/hdds/scm/client/HddsClientUtils.java | 49 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 15 +-
hadoop-hdds/common/pom.xml | 61 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 6 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 25 +-
.../hadoop/hdds/annotation/InterfaceStability.java | 3 -
.../org/apache/hadoop/hdds/cli/GenericCli.java | 12 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 193 +----
.../hadoop/hdds/freon/FakeClusterTopology.java | 92 +++
.../freon/FakeScmBlockLocationProtocolClient.java | 100 +++
.../FakeScmContainerLocationProtocolClient.java | 76 ++
.../apache/hadoop/hdds/freon}/package-info.java | 14 +-
.../java/org/apache/hadoop/hdds/fs/DUFactory.java | 11 +-
.../hdds/fs/DedicatedDiskSpaceUsageFactory.java | 11 +-
.../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 26 +-
.../hdds/function/SupplierWithIOException.java | 19 +-
.../hdds/ratis/ContainerCommandRequestMessage.java | 7 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 32 +-
.../hadoop/hdds/scm/ByteStringConversion.java | 4 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 24 +
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 4 +-
.../hadoop/hdds/scm/container/ContainerInfo.java | 56 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 4 +-
.../hadoop/hdds/scm/net/NodeSchemaManager.java | 4 +-
.../protocol/StorageContainerLocationProtocol.java | 12 +
...inerLocationProtocolClientSideTranslatorPB.java | 42 +-
.../hadoop/hdds/security/x509/SecurityConfig.java | 22 +-
.../hadoop/hdds/tracing/GrpcServerInterceptor.java | 11 +-
.../apache/hadoop/hdds/tracing/TraceAllMethod.java | 8 +-
.../apache/hadoop/hdds/tracing/TracingUtil.java | 71 +-
.../utils/LegacyHadoopConfigurationSource.java | 74 ++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 30 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +-
.../org/apache/hadoop/ozone/OzoneSecurityUtil.java | 30 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
.../org/apache/hadoop/ozone/lock/LockManager.java | 6 +-
.../java/org/apache/hadoop/ozone/package-info.java | 21 +-
.../proto/StorageContainerLocationProtocol.proto | 12 +
.../common/src/main/resources/ozone-default.xml | 82 ++-
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 7 +-
.../hadoop/hdds/conf/TestOzoneConfiguration.java | 18 +-
.../org/apache/hadoop/hdds/fs/TestDUFactory.java | 8 +-
.../fs/TestDedicatedDiskSpaceUsageFactory.java | 8 +-
.../hadoop/hdds/fs/TestSpaceUsageFactory.java | 40 +-
.../hdds/scm/net/TestNetworkTopologyImpl.java | 38 +-
.../hadoop/hdds/scm/net/TestNodeSchemaManager.java | 16 +-
hadoop-hdds/config/pom.xml | 13 +-
.../hdds/conf/ConfigurationReflectionUtil.java | 159 ++++
.../hadoop/hdds/conf/ConfigurationSource.java | 291 ++++++++
.../org/apache/hadoop/hdds/conf/StorageSize.java | 102 +++
.../org/apache/hadoop/hdds/conf/StorageUnit.java | 529 +++++++++++++
.../apache/hadoop/hdds/conf/TimeDurationUtil.java | 154 ++++
hadoop-hdds/container-service/pom.xml | 17 +-
.../hadoop/hdds/freon/FakeRatisFollower.java | 125 ++++
.../apache/hadoop/hdds/freon}/package-info.java | 8 +-
.../hadoop/ozone/HddsDatanodeHttpServer.java | 4 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 4 +-
.../container/common/helpers/ContainerMetrics.java | 4 +-
.../container/common/impl/ChunkLayOutVersion.java | 62 +-
.../ozone/container/common/impl/ContainerData.java | 19 +
.../container/common/impl/HddsDispatcher.java | 87 ++-
.../ozone/container/common/interfaces/Handler.java | 8 +-
.../container/common/report/ReportManager.java | 21 +-
.../container/common/report/ReportPublisher.java | 27 +-
.../common/report/ReportPublisherFactory.java | 32 +-
.../common/statemachine/DatanodeStateMachine.java | 51 +-
.../common/statemachine/EndpointStateMachine.java | 6 +-
.../common/statemachine/SCMConnectionManager.java | 63 +-
.../common/statemachine/StateContext.java | 62 +-
.../CreatePipelineCommandHandler.java | 29 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 6 +-
.../ReplicateContainerCommandHandler.java | 6 +-
.../common/states/datanode/InitDatanodeState.java | 32 +-
.../states/datanode/RunningDatanodeState.java | 6 +-
.../states/endpoint/HeartbeatEndpointTask.java | 10 +-
.../states/endpoint/RegisterEndpointTask.java | 10 +-
.../states/endpoint/VersionEndpointTask.java | 23 +-
.../common/transport/server/XceiverServerGrpc.java | 39 +-
.../server/ratis/ContainerStateMachine.java | 110 ++-
.../transport/server/ratis/XceiverServerRatis.java | 118 +--
.../container/common/utils/ContainerCache.java | 39 +-
.../ozone/container/common/volume/HddsVolume.java | 25 +-
.../container/common/volume/HddsVolumeChecker.java | 4 +-
.../container/common/volume/MutableVolumeSet.java | 13 +-
.../ozone/container/common/volume/VolumeInfo.java | 17 +-
.../container/keyvalue/KeyValueContainer.java | 49 +-
.../container/keyvalue/KeyValueContainerCheck.java | 133 ++--
.../container/keyvalue/KeyValueContainerData.java | 24 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 115 +--
.../container/keyvalue/helpers/BlockUtils.java | 36 +-
.../container/keyvalue/helpers/ChunkUtils.java | 39 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 23 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 6 +-
.../keyvalue/impl/ChunkManagerFactory.java | 4 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 23 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 25 +-
.../keyvalue/interfaces/ChunkManager.java | 1 -
.../background/BlockDeletingService.java | 81 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 30 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 9 +-
.../replication/ContainerStreamingOutput.java | 44 --
.../replication/SimpleContainerDownloader.java | 4 +-
...inerDatanodeProtocolServerSideTranslatorPB.java | 3 +-
.../ozone/container/common/ContainerTestUtils.java | 24 +-
.../ozone/container/common/SCMTestUtils.java | 23 +-
.../container/common/TestBlockDeletingService.java | 80 +-
.../ozone/container/common/TestContainerCache.java | 2 -
.../container/common/TestDatanodeStateMachine.java | 60 +-
.../common/TestKeyValueContainerData.java | 18 +-
.../common/impl/TestContainerDataYaml.java | 21 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 21 +-
.../common/impl/TestContainerPersistence.java | 98 +--
.../container/common/impl/TestContainerSet.java | 23 +-
.../container/common/impl/TestHddsDispatcher.java | 23 +-
.../container/common/interfaces/TestHandler.java | 19 +-
.../container/common/report/TestReportManager.java | 11 +-
.../common/report/TestReportPublisher.java | 4 +-
.../common/report/TestReportPublisherFactory.java | 14 +-
.../TestCloseContainerCommandHandler.java | 21 +-
.../states/endpoint/TestHeartbeatEndpointTask.java | 52 +-
.../container/common/volume/TestHddsVolume.java | 28 +-
.../common/volume/TestHddsVolumeChecker.java | 40 +-
.../volume/TestRoundRobinVolumeChoosingPolicy.java | 22 +-
.../common/volume/TestVolumeSetDiskChecks.java | 16 +-
.../container/keyvalue/ChunkLayoutTestInfo.java | 120 +++
.../keyvalue/TestKeyValueBlockIterator.java | 58 +-
.../container/keyvalue/TestKeyValueContainer.java | 23 +-
.../keyvalue/TestKeyValueContainerCheck.java | 76 +-
.../TestKeyValueContainerMarkUnhealthy.java | 16 +-
.../container/keyvalue/TestKeyValueHandler.java | 164 +++--
.../container/keyvalue/TestTarContainerPacker.java | 18 +-
.../keyvalue/impl/AbstractTestChunkManager.java | 170 +++++
.../keyvalue/impl/CommonChunkManagerTestCases.java | 201 +++++
.../keyvalue/impl/TestBlockManagerImpl.java | 17 +-
.../container/keyvalue/impl/TestChunkManager.java | 325 --------
.../keyvalue/impl/TestChunkManagerDummyImpl.java | 60 ++
.../keyvalue/impl/TestFilePerBlockStrategy.java | 140 ++++
.../keyvalue/impl/TestFilePerChunkStrategy.java | 92 +++
.../container/ozoneimpl/TestOzoneContainer.java | 19 +-
.../replication/TestReplicationSupervisor.java | 32 +-
.../testutils/BlockDeletingServiceTestImpl.java | 16 +-
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 10 +-
hadoop-hdds/docs/archetypes/design.md | 60 ++
hadoop-hdds/docs/content/beyond/Containers.zh.md | 203 +++++
hadoop-hdds/docs/content/concept/Datanodes.md | 4 +-
hadoop-hdds/docs/content/concept/Datanodes.zh.md | 30 +-
hadoop-hdds/docs/content/design/configless.md | 31 +
hadoop-hdds/docs/content/design/decommissioning.md | 17 +-
hadoop-hdds/docs/content/design/gdpr.md | 31 +
hadoop-hdds/docs/content/design/locks.md | 29 +
hadoop-hdds/docs/content/design/multiraft.md | 31 +
.../docs/content/design/namespace-support.md | 29 +
hadoop-hdds/docs/content/design/nfs.md | 31 +
hadoop-hdds/docs/content/design/ofs.md | 33 +
hadoop-hdds/docs/content/design/omha.md | 33 +
.../content/design/ozone-enhancement-proposals.md | 2 +-
.../docs/content/design/ozone-volume-management.md | 181 +++++
hadoop-hdds/docs/content/design/recon1.md | 32 +
hadoop-hdds/docs/content/design/recon2.md | 29 +
hadoop-hdds/docs/content/design/s3gateway.md | 31 +
hadoop-hdds/docs/content/design/scmha.md | 29 +
hadoop-hdds/docs/content/design/tde.md | 29 +
hadoop-hdds/docs/content/design/token.md | 25 +
hadoop-hdds/docs/content/design/trash.md | 25 +
hadoop-hdds/docs/content/design/typesafeconfig.md | 33 +
hadoop-hdds/docs/content/recipe/Prometheus.zh.md | 93 +++
.../{tools/SCMCLI.md => recipe/_index.zh.md} | 15 +-
hadoop-hdds/docs/content/shell/VolumeCommands.md | 6 +-
.../docs/content/shell/VolumeCommands.zh.md | 5 +-
hadoop-hdds/docs/content/tools/Admin.md | 35 +
hadoop-hdds/docs/content/tools/AuditParser.zh.md | 68 ++
.../Datanodes.zh.md => tools/Genconf.zh.md} | 11 +-
hadoop-hdds/docs/content/tools/TestTools.md | 8 +-
.../tools/{TestTools.md => TestTools.zh.md} | 53 +-
hadoop-hdds/docs/content/tools/_index.md | 4 +-
hadoop-hdds/docs/pom.xml | 4 +-
.../ozonedoc/layouts/_default/baseof.html} | 31 +-
.../themes/ozonedoc/layouts/design/section.html | 68 ++
.../themes/ozonedoc/layouts/design/single.html | 47 ++
.../themes/ozonedoc/layouts/partials/sidebar.html | 1 +
hadoop-hdds/framework/pom.xml | 49 +-
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 0
.../hdds/conf/DatanodeRatisServerConfig.java | 0
.../apache/hadoop/hdds/conf/HddsConfServlet.java | 30 +-
.../org/apache/hadoop/hdds/conf}/package-info.java | 5 +-
.../certificates/utils/CertificateSignRequest.java | 23 +-
.../certificates/utils/SelfSignedCertificate.java | 27 +-
.../hdds/security/x509/keys/HDDSKeyGenerator.java | 13 +-
.../org/apache/hadoop/hdds/server}/JsonUtils.java | 8 +-
.../server/OzoneProtocolMessageDispatcher.java | 18 +-
.../org/apache/hadoop/hdds/server/ServerUtils.java | 12 +-
.../hadoop/hdds/server/http/BaseHttpServer.java | 36 +-
.../hadoop/hdds/server/http/FilterInitializer.java | 4 +-
.../apache/hadoop/hdds/server/http/HttpConfig.java | 4 +-
.../hadoop/hdds/server/http/HttpServer2.java | 98 +--
.../server/http/RatisNameRewriteSampleBuilder.java | 17 +-
.../hdds/server/http/StaticUserWebFilter.java | 6 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 39 +-
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 13 +-
.../hadoop/hdds/utils/ProtocolMessageMetrics.java | 42 +-
.../hdds/utils/db/BatchOperationHandler.java | 38 +-
.../hdds/utils/db/DBColumnFamilyDefinition.java | 81 ++
.../apache/hadoop/hdds/utils/db/DBDefinition.java} | 39 +-
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 1 -
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 18 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 56 +-
.../apache/hadoop/hdds/utils/db/RDBMetrics.java | 13 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 2 +-
.../apache/hadoop/hdds/server/TestJsonUtils.java | 37 +-
.../apache/hadoop/hdds/server/TestServerUtils.java | 21 +-
.../hdds/server/http/TestBaseHttpServer.java | 4 +-
.../hdds/server/http/TestRatisNameRewrite.java | 10 +
.../hadoop/hdds/utils/TestMetadataStore.java | 13 +-
.../hadoop/hdds/utils/TestRocksDBStoreMBean.java | 23 +-
hadoop-hdds/hadoop-dependency-client/README.md | 60 ++
hadoop-hdds/hadoop-dependency-client/pom.xml | 282 +++++++
.../{tools => hadoop-dependency-server}/pom.xml | 49 +-
.../{client => hadoop-dependency-test}/pom.xml | 26 +-
hadoop-hdds/pom.xml | 34 +-
hadoop-hdds/server-scm/pom.xml | 26 +-
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 35 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 33 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 34 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 41 +-
.../hdds/scm/container/ContainerManager.java | 6 +-
.../hdds/scm/container/ContainerReportHandler.java | 57 +-
.../hdds/scm/container/ContainerStateManager.java | 61 +-
.../hdds/scm/container/ReplicationManager.java | 9 +-
.../hdds/scm/container/SCMContainerManager.java | 145 ++--
.../ContainerPlacementPolicyFactory.java | 11 +-
.../algorithms/SCMContainerPlacementCapacity.java | 4 +-
.../algorithms/SCMContainerPlacementRackAware.java | 4 +-
.../algorithms/SCMContainerPlacementRandom.java | 4 +-
.../container/placement/metrics/ContainerStat.java | 2 +-
.../hadoop/hdds/scm/metadata/ContainerIDCodec.java | 48 ++
.../hdds/scm/metadata/ContainerInfoCodec.java | 47 ++
.../hadoop/hdds/scm/metadata/PipelineCodec.java | 56 ++
.../hadoop/hdds/scm/metadata/PipelineIDCodec.java | 45 ++
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 98 +++
.../hadoop/hdds/scm/metadata/SCMMetadataStore.java | 27 +-
.../hdds/scm/metadata/SCMMetadataStoreRDBImpl.java | 113 +--
.../hadoop/hdds/scm/node/NewNodeHandler.java | 15 +-
.../hadoop/hdds/scm/node/NodeStateManager.java | 55 +-
.../scm/node/NonHealthyToHealthyNodeHandler.java | 4 +-
.../hadoop/hdds/scm/node/StaleNodeHandler.java | 4 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 15 +-
.../hdds/scm/pipeline/PipelineActionHandler.java | 63 +-
.../hadoop/hdds/scm/pipeline/PipelineFactory.java | 6 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 140 ++--
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 75 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 76 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 11 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 178 +++--
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 12 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 15 +-
...lockLocationProtocolServerSideTranslatorPB.java | 4 +-
...inerLocationProtocolServerSideTranslatorPB.java | 27 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 4 +-
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 4 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 11 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 4 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 60 +-
.../hadoop/hdds/scm/safemode/SafeModeHandler.java | 23 +-
.../hadoop/hdds/scm/safemode/SafeModePrecheck.java | 4 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 6 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 185 +++--
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 21 +-
.../hdds/scm/server/StorageContainerManager.java | 88 ++-
.../server/StorageContainerManagerHttpServer.java | 8 +-
.../apache/hadoop/hdds/scm/TestHddsServerUtil.java | 18 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 42 +-
.../scm/TestStorageContainerManagerHttpServer.java | 6 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 37 +-
.../container/TestCloseContainerEventHandler.java | 38 +-
.../scm/container/TestContainerReportHandler.java | 4 +-
.../TestIncrementalContainerReportHandler.java | 4 +-
.../hdds/scm/container/TestReplicationManager.java | 34 +-
.../scm/container/TestSCMContainerManager.java | 65 +-
.../scm/container/TestUnknownContainerReport.java | 145 ++++
.../algorithms/TestContainerPlacementFactory.java | 19 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 4 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../hdds/scm/node/TestContainerPlacement.java | 65 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 14 +-
.../scm/pipeline/MockRatisPipelineProvider.java | 20 +-
.../scm/pipeline/TestPipelineActionHandler.java | 69 ++
.../scm/pipeline/TestPipelinePlacementPolicy.java | 93 ++-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 178 ++++-
.../scm/pipeline/TestSimplePipelineProvider.java | 2 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 38 +-
.../TestOneReplicaPipelineSafeModeRule.java | 19 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 179 ++++-
.../hdds/scm/safemode/TestSafeModeHandler.java | 71 +-
.../ozone/container/common/TestEndPoint.java | 7 +-
.../placement/TestContainerPlacement.java | 32 +-
.../upgrade => hadoop-hdds/test-utils}/pom.xml | 50 +-
.../org/apache/hadoop/test/GenericTestUtils.java | 365 +++++++++
.../org/apache/hadoop/test/LambdaTestUtils.java | 814 +++++++++++++++++++++
.../apache/hadoop/test/TimedOutTestsListener.java | 183 +++++
.../java/org/apache/hadoop/test}/package-info.java | 7 +-
hadoop-hdds/tools/pom.xml | 13 +-
.../hdds/scm/cli/ContainerOperationClient.java | 24 +-
.../hdds/scm/cli/ReplicationManagerCommands.java | 21 +-
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 114 ---
.../hadoop/hdds/scm/cli/SafeModeCommands.java | 16 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 26 +-
.../hdds/scm/cli/container/CloseSubcommand.java | 4 +-
.../hdds/scm/cli/container/ContainerCommands.java | 32 +-
.../hdds/scm/cli/container/DeleteSubcommand.java | 3 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 20 +-
.../hdds/scm/cli/container/ListSubcommand.java | 21 +-
.../hdds/scm/cli/container/WithScmClient.java | 13 +-
.../hdds/scm/cli/datanode/DatanodeCommands.java | 22 +-
.../hdds/scm/cli/pipeline/PipelineCommands.java | 22 +-
hadoop-ozone/Jenkinsfile | 116 ---
hadoop-ozone/client/pom.xml | 9 +-
.../apache/hadoop/ozone/client/ObjectStore.java | 6 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 8 +-
.../apache/hadoop/ozone/client/OzoneClient.java | 4 +-
.../hadoop/ozone/client/OzoneClientFactory.java | 69 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 18 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 36 +-
.../client/io/BlockOutputStreamEntryPool.java | 8 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 16 +-
.../hadoop/ozone/client/rpc/OzoneKMSUtil.java | 32 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 91 ++-
.../hadoop/ozone/client/TestHddsClientUtils.java | 41 +-
hadoop-ozone/common/pom.xml | 29 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 166 +++--
.../apache/hadoop/ozone/conf/OMClientConfig.java | 67 ++
.../apache/hadoop/ozone/conf}/package-info.java | 11 +-
.../apache/hadoop/ozone/freon/OzoneGetConf.java | 27 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 5 +-
.../hadoop/ozone/om/exceptions/OMException.java | 2 +
.../ozone/om/exceptions/OMReplayException.java | 16 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 105 ++-
.../apache/hadoop/ozone/om/helpers/DBUpdates.java | 58 ++
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 2 +
.../hadoop/ozone/om/helpers/OzoneFileStatus.java | 116 ++-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 24 +-
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 4 +-
.../ozone/om/protocol/OzoneManagerProtocol.java | 20 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 127 ++--
.../ozone/security/OzoneTokenIdentifier.java | 22 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 7 +-
.../common/src/main/proto/FSProtos.proto | 2 +-
.../src/main/proto/OzoneManagerProtocol.proto | 6 +
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 125 +---
hadoop-ozone/csi/pom.xml | 4 +-
hadoop-ozone/datanode/pom.xml | 6 +-
hadoop-ozone/dev-support/checks/author.sh | 6 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 14 +-
.../intellij/runConfigurations/OzoneShell.xml | 6 +-
hadoop-ozone/dist/pom.xml | 16 +-
.../main/compose/ozone-hdfs/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-mr/common-config | 2 +-
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 1 +
.../ozone-mr/hadoop27/{test.sh => run-test.sh} | 0
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 1 +
.../src/main/compose/ozone-mr/hadoop31/test.sh | 6 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-om-ha/.ssh/id_rsa | 15 -
.../src/main/compose/ozone-om-ha/.ssh/id_rsa.pub | 15 -
.../src/main/compose/ozone-om-ha/docker-config | 3 +-
.../ozone-om-ha/{run.sh => test_disabled.sh} | 0
.../dist/src/main/compose/ozone-topology/.env | 2 +-
.../compose/ozone-topology/docker-compose.yaml | 3 +-
.../src/main/compose/ozone-topology/docker-config | 2 +-
.../src/main/compose/ozone-topology/hdds-3084.sh | 56 --
.../dist/src/main/compose/ozone-topology/test.sh | 20 +-
.../src/main/compose/ozone/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
.../main/compose/ozoneblockade/docker-compose.yaml | 1 +
.../compose/ozones3-haproxy/docker-compose.yaml | 1 +
.../main/compose/ozonescripts/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 12 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 1 +
.../src/main/compose/ozonesecure-mr/docker-config | 2 +-
.../main/compose/ozonesecure/docker-compose.yaml | 1 +
.../src/main/compose/ozonesecure/docker-config | 1 +
.../dist/src/main/compose/ozonesecure/test.sh | 2 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 2 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 24 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 8 -
.../licenses/LICENSE-org.openjdk.jmh-jmh-core.txt | 347 ---------
...SE-org.openjdk.jmh-jmh-generator-annprocess.txt | 347 ---------
.../smoketest/{scmcli => admincli}/datanode.robot | 3 +-
.../smoketest/{scmcli => admincli}/pipeline.robot | 5 +-
.../main/smoketest/auditparser/auditparser.robot | 1 +
.../dist/src/main/smoketest/basic/basic.robot | 1 +
.../src/main/smoketest/basic/ozone-shell.robot | 10 +-
.../dist/src/main/smoketest/createbucketenv.robot | 2 +-
.../dist/src/main/smoketest/createmrenv.robot | 4 +-
.../dist/src/main/smoketest/env-compose.robot | 2 +-
.../dist/src/main/smoketest/freon/freon.robot | 1 +
.../dist/src/main/smoketest/gdpr/gdpr.robot | 1 +
.../dist/src/main/smoketest/mapreduce.robot | 4 +-
.../main/smoketest/om-ratis/testOMAdminCmd.robot | 2 +-
.../dist/src/main/smoketest/omha/testOMHA.robot | 45 +-
.../src/main/smoketest/ozonefs/hadoopo3fs.robot | 5 +-
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 5 +
.../dist/src/main/smoketest/recon/recon-api.robot | 4 +
.../src/main/smoketest/s3/MultipartUpload.robot | 1 +
.../dist/src/main/smoketest/s3/awss3.robot | 1 +
.../dist/src/main/smoketest/s3/bucketcreate.robot | 1 +
.../dist/src/main/smoketest/s3/buckethead.robot | 1 +
.../dist/src/main/smoketest/s3/bucketlist.robot | 1 +
.../dist/src/main/smoketest/s3/objectcopy.robot | 1 +
.../dist/src/main/smoketest/s3/objectdelete.robot | 1 +
.../src/main/smoketest/s3/objectmultidelete.robot | 1 +
.../dist/src/main/smoketest/s3/objectputget.robot | 1 +
.../dist/src/main/smoketest/s3/webui.robot | 1 +
.../main/smoketest/security/ozone-secure-fs.robot | 1 +
.../main/smoketest/security/ozone-secure-s3.robot | 1 +
.../smoketest/security/ozone-secure-token.robot | 76 ++
.../smoketest/topology/{scmcli.robot => cli.robot} | 5 +-
.../src/main/smoketest/topology/loaddata.robot | 1 +
.../src/main/smoketest/topology/readdata.robot | 1 +
.../src/shell/conf/ozone-shell-log4j.properties | 1 +
.../dist/src/shell/hdds/hadoop-functions.sh | 63 ++
hadoop-ozone/dist/src/shell/ozone/ozone | 68 +-
hadoop-ozone/dist/src/test/shell/gc_opts.bats | 44 ++
.../fault-injection-test/mini-chaos-tests/pom.xml | 14 +-
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 252 +++++--
.../ozone/MiniOzoneDatanodeChaosCluster.java | 57 ++
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 14 +-
.../hadoop/ozone/MiniOzoneOMChaosCluster.java | 132 ++++
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 46 +-
.../ozone/loadgenerators/AgedLoadGenerator.java | 20 +-
.../hadoop/ozone/loadgenerators/DataBuffer.java | 1 +
.../loadgenerators/FilesystemLoadGenerator.java | 15 +-
.../hadoop/ozone/loadgenerators/LoadExecutors.java | 25 +-
.../hadoop/ozone/loadgenerators/LoadGenerator.java | 13 +-
.../ozone/loadgenerators/RandomLoadGenerator.java | 10 +-
...adGenerator.java => ReadOnlyLoadGenerator.java} | 49 +-
.../org/apache/hadoop/ozone/utils/LoadBucket.java | 17 +-
.../fault-injection-test/network-tests/pom.xml | 2 +-
hadoop-ozone/fault-injection-test/pom.xml | 4 +-
hadoop-ozone/insight/pom.xml | 30 +-
.../apache/hadoop/ozone/insight/LogSubcommand.java | 2 +
hadoop-ozone/integration-test/pom.xml | 16 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 112 ++-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 7 -
.../ozone/contract/ITestOzoneContractCreate.java | 5 +-
.../ozone/contract/ITestOzoneContractDelete.java | 5 +-
.../ozone/contract/ITestOzoneContractDistCp.java | 5 +-
.../contract/ITestOzoneContractGetFileStatus.java | 5 +-
.../fs/ozone/contract/ITestOzoneContractMkdir.java | 5 +-
.../fs/ozone/contract/ITestOzoneContractOpen.java | 5 +-
.../ozone/contract/ITestOzoneContractRename.java | 5 +-
.../ozone/contract/ITestOzoneContractRootDir.java | 6 +-
.../fs/ozone/contract/ITestOzoneContractSeek.java | 5 +-
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 6 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 33 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 12 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 157 +++-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 3 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 45 +-
.../ozone/TestStorageContainerManagerHelper.java | 10 +-
.../ozone/client/CertificateClientTestImpl.java | 24 +-
.../ozone/client/rpc/Test2BlockOutputStream.java | 173 +++++
.../TestContainerStateMachineFailureOnRead.java | 16 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 24 +-
.../client/rpc/TestFailureHandlingByClient.java | 23 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 23 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 23 +-
.../ozone/dn/ratis/TestDnRatisLogParser.java | 21 +-
.../ozone/freon/TestOzoneClientKeyGenerator.java | 13 +-
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 8 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 37 +-
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 116 ++-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 27 +-
.../ozone/om/TestOzoneManagerListVolumes.java | 238 ++++++
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 2 +-
.../ozone/om/parser/TestOMRatisLogParser.java | 5 +-
.../{ozShell => shell}/TestOzoneDatanodeShell.java | 2 +-
.../ozone/{ozShell => shell}/TestOzoneShellHA.java | 6 +-
.../ozone/{ozShell => shell}/TestS3Shell.java | 20 +-
hadoop-ozone/ozone-manager/pom.xml | 17 +-
.../apache/hadoop/ozone/om/KeyDeletingService.java | 4 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 70 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 91 ++-
.../apache/hadoop/ozone/om/OMMetadataManager.java | 0
.../java/org/apache/hadoop/ozone/om/OMStorage.java | 4 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 2 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 173 +++--
.../hadoop/ozone/om/OzoneManagerHttpServer.java | 10 +-
.../hadoop/ozone/om/S3SecretManagerImpl.java | 0
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 22 +-
.../hadoop/ozone/om/codec/OmBucketInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 0
.../ozone/om/codec/OmMultipartKeyInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmPrefixInfoCodec.java | 0
.../hadoop/ozone/om/codec/OmVolumeArgsCodec.java | 0
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 0
.../hadoop/ozone/om/codec/S3SecretValueCodec.java | 0
.../ozone/om/codec/TokenIdentifierCodec.java | 0
.../hadoop/ozone/om/codec/UserVolumeInfoCodec.java | 0
.../apache/hadoop/ozone/om/codec/package-info.java | 0
.../ozone/om/ratis/OzoneManagerRatisServer.java | 37 +-
.../om/request/bucket/OMBucketCreateRequest.java | 4 +
.../om/request/file/OMDirectoryCreateRequest.java | 42 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 13 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 60 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 7 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 80 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 13 +-
.../S3InitiateMultipartUploadRequest.java | 6 -
.../S3MultipartUploadCommitPartRequest.java | 69 +-
.../S3MultipartUploadCompleteRequest.java | 78 +-
.../om/request/volume/OMVolumeCreateRequest.java | 20 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 21 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 26 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 2 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 43 +-
.../S3MultipartUploadCommitPartResponse.java | 59 +-
.../S3MultipartUploadCompleteResponse.java | 48 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 21 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 6 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 13 +-
.../protocolPB/OzoneManagerRequestHandler.java | 26 +-
.../hadoop/ozone/security/AWSV4AuthValidator.java | 0
.../security/OzoneBlockTokenSecretManager.java | 0
.../OzoneDelegationTokenSecretManager.java | 99 ++-
.../hadoop/ozone/security/OzoneSecretKey.java | 0
.../hadoop/ozone/security/OzoneSecretManager.java | 0
.../hadoop/ozone/security/OzoneSecretStore.java | 0
.../hadoop/ozone/security}/package-info.java | 7 +-
.../web/ozShell/bucket/AddAclBucketHandler.java | 105 ---
.../web/ozShell/bucket/CreateBucketHandler.java | 107 ---
.../web/ozShell/bucket/DeleteBucketHandler.java | 64 --
.../web/ozShell/bucket/GetAclBucketHandler.java | 85 ---
.../web/ozShell/bucket/InfoBucketHandler.java | 68 --
.../web/ozShell/bucket/ListBucketHandler.java | 105 ---
.../web/ozShell/bucket/RemoveAclBucketHandler.java | 106 ---
.../web/ozShell/bucket/SetAclBucketHandler.java | 100 ---
.../ozone/web/ozShell/keys/AddAclKeyHandler.java | 109 ---
.../ozone/web/ozShell/keys/DeleteKeyHandler.java | 70 --
.../ozone/web/ozShell/keys/GetAclKeyHandler.java | 89 ---
.../ozone/web/ozShell/keys/GetKeyHandler.java | 113 ---
.../ozone/web/ozShell/keys/InfoKeyHandler.java | 78 --
.../ozone/web/ozShell/keys/ListKeyHandler.java | 113 ---
.../ozone/web/ozShell/keys/PutKeyHandler.java | 129 ----
.../web/ozShell/keys/RemoveAclKeyHandler.java | 109 ---
.../ozone/web/ozShell/keys/SetAclKeyHandler.java | 102 ---
.../ozone/web/ozShell/s3/GetS3SecretHandler.java | 63 --
.../web/ozShell/token/CancelTokenHandler.java | 76 --
.../ozone/web/ozShell/token/GetTokenHandler.java | 81 --
.../ozone/web/ozShell/token/PrintTokenHandler.java | 71 --
.../ozone/web/ozShell/token/RenewTokenHandler.java | 79 --
.../web/ozShell/volume/AddAclVolumeHandler.java | 103 ---
.../web/ozShell/volume/CreateVolumeHandler.java | 103 ---
.../web/ozShell/volume/DeleteVolumeHandler.java | 62 --
.../web/ozShell/volume/GetAclVolumeHandler.java | 80 --
.../web/ozShell/volume/ListVolumeHandler.java | 111 ---
.../web/ozShell/volume/RemoveAclVolumeHandler.java | 103 ---
.../web/ozShell/volume/SetAclVolumeHandler.java | 106 ---
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 13 +-
.../org/apache/hadoop/ozone/om/TestOMStorage.java | 9 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 26 +-
.../ozone/om/TestOzoneManagerHttpServer.java | 21 +-
.../om/codec/TestOmMultipartKeyInfoCodec.java | 0
.../ozone/om/codec/TestOmPrefixInfoCodec.java | 0
.../ozone/om/codec/TestS3SecretValueCodec.java | 0
.../apache/hadoop/ozone/om/codec/package-info.java | 0
.../request/bucket/TestOMBucketCreateRequest.java | 5 +
.../om/request/key/TestOMKeyCommitRequest.java | 42 ++
.../s3/bucket/TestS3BucketCreateRequest.java | 4 +-
.../request/volume/TestOMVolumeCreateRequest.java | 5 +
.../volume/TestOMVolumeSetOwnerRequest.java | 49 ++
.../om/response/key/TestOMKeyCommitResponse.java | 14 +-
.../ozone/security/TestAWSV4AuthValidator.java | 0
.../TestOzoneDelegationTokenSecretManager.java | 14 +-
.../ozone/security/TestOzoneTokenIdentifier.java | 41 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 17 +-
hadoop-ozone/ozonefs-lib-legacy/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 36 +-
.../java/org/apache/hadoop/fs/ozone/BasicOzFs.java | 12 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 13 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 20 +-
.../org/apache/hadoop/fs/ozone/O3fsDtFetcher.java | 8 +-
.../main/java/org/apache/hadoop/fs/ozone/OzFs.java | 12 +-
.../hadoop/fs/ozone/OzoneClientAdapterImpl.java | 5 +-
.../apache/hadoop/fs/ozone/OzoneFileSystem.java | 10 +-
.../org/apache/hadoop/fs/ozone/OzoneFsShell.java | 5 +-
.../fs/ozone/TestOzoneFileSystemWithMocks.java | 31 +-
.../hadoop/fs/ozone/TestReadWriteStatistics.java | 41 +-
hadoop-ozone/pom.xml | 29 +-
hadoop-ozone/recon-codegen/pom.xml | 7 +-
.../ozone/recon/codegen/JooqCodeGenerator.java | 2 +-
.../recon/codegen/ReconSchemaGenerationModule.java | 2 +
.../recon/schema/ContainerSchemaDefinition.java | 87 +++
.../recon/schema/ReconTaskSchemaDefinition.java | 2 +
.../ozone/recon/schema/StatsSchemaDefinition.java | 2 +
.../recon/schema/UtilizationSchemaDefinition.java | 19 +-
hadoop-ozone/recon/pom.xml | 33 +-
.../hadoop/ozone/recon/ConfigurationProvider.java | 5 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 51 +-
.../hadoop/ozone/recon/ReconSchemaManager.java | 4 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 10 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 54 +-
.../ozone/recon/api/ClusterStateEndpoint.java | 36 +-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 29 +-
.../hadoop/ozone/recon/api/types/KeysResponse.java | 61 +-
.../recon/api/types/MissingContainerMetadata.java | 14 +-
.../ozone/recon/fsck/MissingContainerTask.java | 22 +-
.../recon/metrics/OzoneManagerSyncMetrics.java | 133 ++++
.../hadoop/ozone/recon/metrics}/package-info.java | 16 +-
.../recon/persistence/ContainerSchemaManager.java | 104 +++
.../recon/recovery/ReconOMMetadataManager.java | 6 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 10 +
.../ozone/recon/scm/ReconContainerManager.java | 55 +-
.../hadoop/ozone/recon/scm/ReconDBDefinition.java} | 31 +-
.../recon/scm/ReconDatanodeProtocolServer.java | 4 +-
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 17 +-
.../ozone/recon/scm/ReconPipelineFactory.java | 4 +-
.../ozone/recon/scm/ReconPipelineManager.java | 34 +-
.../recon/scm/ReconPipelineReportHandler.java | 4 +-
.../scm/ReconStorageContainerManagerFacade.java | 46 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 10 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 76 +-
.../webapps/recon/ozone-recon-web/api/db.json | 74 +-
.../webapps/recon/ozone-recon-web/api/routes.json | 3 +-
.../recon/ozone-recon-web/src/utils/common.tsx | 9 +-
.../views/MissingContainers/MissingContainers.less | 4 +
.../views/MissingContainers/MissingContainers.tsx | 99 ++-
.../src/views/Overview/Overview.tsx | 12 +-
...erTest.java => OMMetadataManagerTestUtils.java} | 60 +-
.../hadoop/ozone/recon/ReconTestInjector.java | 337 +++++++++
.../apache/hadoop/ozone/recon/TestReconUtils.java | 45 +-
.../ozone/recon/api/TestContainerEndpoint.java | 170 ++---
.../hadoop/ozone/recon/api/TestEndpoints.java | 141 ++--
.../ozone/recon/api/TestTaskStatusService.java | 23 +-
.../ozone/recon/fsck/TestMissingContainerTask.java | 40 +-
...tabaseTest.java => AbstractReconSqlDBTest.java} | 131 +++-
.../TestReconInternalSchemaDefinition.java | 26 +-
.../persistence/TestStatsSchemaDefinition.java | 25 +-
.../TestUtilizationSchemaDefinition.java | 37 +-
.../scm/AbstractReconContainerManagerTest.java | 38 +-
.../ozone/recon/scm/TestReconContainerManager.java | 2 +-
.../ozone/recon/scm/TestReconPipelineManager.java | 37 +-
.../recon/scm/TestReconPipelineReportHandler.java | 2 +-
.../impl/TestContainerDBServiceProviderImpl.java | 56 +-
.../impl/TestOzoneManagerServiceProviderImpl.java | 64 +-
.../recon/tasks/TestContainerKeyMapperTask.java | 71 +-
.../ozone/recon/tasks/TestFileSizeCountTask.java | 16 +-
.../recon/tasks/TestReconTaskControllerImpl.java | 28 +-
.../recon/types/GuiceInjectorUtilsForTests.java | 136 ----
hadoop-ozone/s3gateway/pom.xml | 36 +-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 2 +
.../hadoop/ozone/s3/S3GatewayHttpServer.java | 4 +-
.../org/apache/hadoop/ozone/s3/TracingFilter.java | 82 +++
hadoop-ozone/tools/pom.xml | 31 +-
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 58 +-
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 12 +-
.../hadoop/ozone/debug/ChunkDataNodeDetails.java} | 31 +-
.../apache/hadoop/ozone/debug/ChunkDetails.java} | 44 +-
.../apache/hadoop/ozone/debug/ChunkKeyHandler.java | 171 +++++
.../hadoop/ozone/debug/ContainerChunkInfo.java | 93 +++
.../org/apache/hadoop/ozone/debug/OzoneDebug.java | 49 +-
.../apache/hadoop/ozone/debug}/package-info.java | 7 +-
.../hadoop/ozone/freon/BaseAppendLogGenerator.java | 10 +
.../hadoop/ozone/freon/BaseFreonGenerator.java | 17 +-
.../freon/FollowerAppendLogEntryGenerator.java | 8 -
.../java/org/apache/hadoop/ozone/freon/Freon.java | 4 +-
.../apache/hadoop/ozone/freon/FreonHttpServer.java | 4 +-
.../ozone/freon/HadoopNestedDirGenerator.java | 113 +++
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 273 +++++++
.../hadoop/ozone/freon/RandomKeyGenerator.java | 27 +-
.../ozone/genesis/BenchMarkContainerStateMap.java | 8 +-
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 5 +-
.../ozone/genesis/BenchMarkOzoneManager.java | 11 +-
.../apache/hadoop/ozone/genesis/BenchMarkSCM.java | 11 +-
.../ozone/genesis/BenchmarkChunkManager.java | 33 +-
.../apache/hadoop/ozone/genesis/GenesisUtil.java | 58 +-
.../org/apache/hadoop/ozone/shell/Handler.java | 133 ++++
.../org/apache/hadoop/ozone/shell/ListOptions.java | 58 ++
.../apache/hadoop/ozone/shell}/OzoneAddress.java | 45 +-
.../org/apache/hadoop/ozone/shell}/OzoneShell.java | 22 +-
.../java/org/apache/hadoop/ozone/shell}/Shell.java | 27 +-
.../hadoop/ozone/shell/StoreTypeOption.java} | 36 +-
.../apache/hadoop/ozone/shell/acl/AclHandler.java | 62 ++
.../apache/hadoop/ozone/shell/acl/AclOption.java | 91 +++
.../hadoop/ozone/shell/acl/GetAclHandler.java} | 24 +-
.../hadoop/ozone/shell/acl}/package-info.java | 4 +-
.../ozone/shell/bucket/AddAclBucketHandler.java} | 36 +-
.../hadoop/ozone/shell}/bucket/BucketCommands.java | 4 +-
.../hadoop/ozone/shell/bucket/BucketHandler.java} | 23 +-
.../hadoop/ozone/shell/bucket/BucketUri.java | 44 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 88 +++
.../ozone/shell/bucket/DeleteBucketHandler.java} | 33 +-
.../ozone/shell/bucket/GetAclBucketHandler.java} | 28 +-
.../ozone/shell/bucket/InfoBucketHandler.java} | 32 +-
.../ozone/shell/bucket/ListBucketHandler.java | 68 ++
.../shell/bucket/RemoveAclBucketHandler.java} | 36 +-
.../ozone/shell/bucket/SetAclBucketHandler.java} | 37 +-
.../hadoop/ozone/shell/bucket}/package-info.java | 4 +-
.../hadoop/ozone/shell/keys/AddAclKeyHandler.java} | 36 +-
.../hadoop/ozone/shell/keys/DeleteKeyHandler.java | 50 ++
.../hadoop/ozone/shell/keys/GetAclKeyHandler.java} | 28 +-
.../hadoop/ozone/shell/keys/GetKeyHandler.java | 98 +++
.../hadoop/ozone/shell/keys/InfoKeyHandler.java | 58 ++
.../hadoop/ozone/shell}/keys/KeyCommands.java | 4 +-
.../hadoop/ozone/shell/keys/KeyHandler.java} | 23 +-
.../org/apache/hadoop/ozone/shell/keys/KeyUri.java | 44 +-
.../hadoop/ozone/shell/keys/ListKeyHandler.java | 78 ++
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 108 +++
.../ozone/shell/keys/RemoveAclKeyHandler.java} | 36 +-
.../hadoop/ozone/shell}/keys/RenameKeyHandler.java | 45 +-
.../hadoop/ozone/shell/keys/SetAclKeyHandler.java} | 37 +-
.../hadoop/ozone/shell/keys}/package-info.java | 4 +-
.../apache/hadoop/ozone/shell}/package-info.java | 7 +-
.../hadoop/ozone/shell/s3/GetS3SecretHandler.java} | 30 +-
.../hadoop/ozone/shell}/s3/S3BucketMapping.java | 42 +-
.../apache/hadoop/ozone/shell}/s3/S3Handler.java | 30 +-
.../org/apache/hadoop/ozone/shell}/s3/S3Shell.java | 17 +-
.../hadoop/ozone/shell}/s3/package-info.java | 2 +-
.../ozone/shell/token/CancelTokenHandler.java} | 29 +-
.../hadoop/ozone/shell/token/GetTokenHandler.java | 72 ++
.../ozone/shell/token/PrintTokenHandler.java} | 46 +-
.../ozone/shell/token/RenewTokenHandler.java} | 29 +-
.../hadoop/ozone/shell/token/RenewerOption.java} | 32 +-
.../hadoop/ozone/shell}/token/TokenCommands.java | 5 +-
.../hadoop/ozone/shell/token/TokenHandler.java | 54 ++
.../hadoop/ozone/shell/token/TokenOption.java | 56 ++
.../hadoop/ozone/shell}/token/package-info.java | 6 +-
.../ozone/shell/volume/AddAclVolumeHandler.java} | 36 +-
.../ozone/shell/volume/CreateVolumeHandler.java | 74 ++
.../ozone/shell/volume/DeleteVolumeHandler.java} | 30 +-
.../ozone/shell/volume/GetAclVolumeHandler.java} | 28 +-
.../ozone/shell}/volume/InfoVolumeHandler.java | 36 +-
.../ozone/shell/volume/ListVolumeHandler.java | 99 +++
.../shell/volume/RemoveAclVolumeHandler.java} | 36 +-
.../ozone/shell/volume/SetAclVolumeHandler.java} | 37 +-
.../ozone/shell}/volume/UpdateVolumeHandler.java | 46 +-
.../hadoop/ozone/shell}/volume/VolumeCommands.java | 4 +-
.../hadoop/ozone/shell/volume/VolumeHandler.java} | 23 +-
.../hadoop/ozone/shell/volume/VolumeUri.java | 44 +-
.../hadoop/ozone/shell/volume}/package-info.java | 4 +-
.../hadoop/ozone/shell}/TestOzoneAddress.java | 2 +-
.../apache/hadoop/ozone/shell}/package-info.java | 2 +-
hadoop-ozone/upgrade/pom.xml | 9 +-
pom.xml | 47 +-
763 files changed, 19402 insertions(+), 11091 deletions(-)
diff --cc hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
index 32a9a94,6184b42..ae1f558
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
@@@ -14,8 -14,8 +14,9 @@@
# See the License for the specific language governing permissions and
# limitations under the License.
+CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
+ OZONE-SITE.XML_ozone.om.volume.listall.allowed=false
OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
diff --cc hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
index adc6e0f,fcc7809..e6ab0d3
--- a/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
@@@ -14,51 -14,35 +14,56 @@@
# limitations under the License.
*** Settings ***
-Documentation Ozonefs test
+Documentation Ozonefs test covering both o3fs and ofs
Library OperatingSystem
Resource ../commonlib.robot
+ Test Timeout 5 minutes
*** Variables ***
+${OfsBucket1} om:9862/fstest/bucket1
+${O3fsBucket1} bucket1.fstest/
+${O3Bucket1} o3://om/fstest/bucket1
+${O3Bucket2} o3://om/fstest/bucket2
+${O3Bucket3} o3://om/fstest2/bucket3
+${OfsNonExistBucket} om:9862/abc/def
+${O3fsNonExistBucket} def.abc/
+${NonExistVolume} abc
*** Test Cases ***
-Create volume and bucket
+Create volume and bucket for Ozone file System test
Execute ozone sh volume create o3://om/fstest --quota 100TB
+ Execute ozone sh bucket create ${O3Bucket1}
+ Execute ozone sh bucket create ${O3Bucket2}
+
Execute ozone sh volume create o3://om/fstest2 --quota 100TB
- Execute ozone sh bucket create o3://om/fstest/bucket1
- Execute ozone sh bucket create o3://om/fstest/bucket2
- Execute ozone sh bucket create o3://om/fstest2/bucket3
+ Execute ozone sh bucket create ${O3Bucket3}
+
-Check volume from ozonefs
+Check volume from o3fs
+ ${result} = Execute ozone sh volume list
+ Should contain ${result} fstest
+ Should contain ${result} fstest2
+ Should Match Regexp ${result} "admin" : "(hadoop|testuser\/scm@EXAMPLE\.COM)"
- ${result} = Execute ozone fs -ls o3fs://bucket1.fstest/
+ ${result} = Execute ozone fs -ls o3fs://${O3fsBucket1}
+
+Test ozone shell with ofs
+ Test ozone shell with scheme ofs om:9862/fstest/bucket1 om:9862/fstest/bucket2 om:9862/fstest2/bucket3 om:9862/abc/def
-Run ozoneFS tests
- Execute ozone fs -mkdir -p o3fs://bucket1.fstest/testdir/deep
- ${result} = Execute ozone sh key list o3://om/fstest/bucket1 | jq -r '.name'
+Test ozone shell with o3fs
+ Test ozone shell with scheme o3fs bucket1.fstest/ bucket2.fstest/ bucket3.fstest2/ def.abc/
+
+*** Keywords ***
+Test ozone shell with scheme
+ [arguments] ${scheme} ${testBucket1} ${testBucket2} ${testBucket3} ${nonExistBucket}
+
+ ${result} = Execute ozone fs -ls ${scheme}://${testBucket1}
+ Execute ozone fs -mkdir -p ${scheme}://${testBucket1}/testdir/deep
+ ${result} = Execute ozone sh key list ${O3Bucket1} | jq -r '.name'
Should contain ${result} testdir/deep
- Execute ozone fs -copyFromLocal NOTICE.txt o3fs://bucket1.fstest/testdir/deep/
- ${result} = Execute ozone sh key list o3://om/fstest/bucket1 | jq -r '.name'
+
+ Execute ozone fs -copyFromLocal NOTICE.txt ${scheme}://${testBucket1}/testdir/deep/
+ ${result} = Execute ozone sh key list ${O3Bucket1} | jq -r '.name'
Should contain ${result} NOTICE.txt
${result} = Execute ozone sh key info o3://om/fstest/bucket1/testdir/deep/NOTICE.txt | jq -r '.replicationFactor'
Should Be Equal ${result} 3
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org