You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by sa...@apache.org on 2020/07/22 13:00:10 UTC
[hadoop-ozone] branch ozone-0.6.0 updated (ca9b368 -> 5ac4073)
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from ca9b368 HDDS-3926. OM Token Identifier table should use in-house serialization. (#1182)
new c9e5a7f HDDS-3914. Remove LevelDB configuration option for DN Metastore (#1166)
new 4aa6087 HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen… (#1162)
new f58edcf HDDS-3937. Update jquery to v3.5.1 (#1177)
new 9c9daa1 HDDS-3941. Enable core dump when crash in C++ (#1186)
new 16d8a3a HDDS-3920. Too many redudant replications due to fail to get node's a… (#1163)
new 29e26b0 HDDS-3861. Fix handlePipelineFailure throw exception if role is follower (#1122)
new 3a75437 HDDS-3789. Fix TestOzoneRpcClientAbstract#testDeletedKeyForGDPR. (#1198)
new 2e529a9 HDDS-3062. Fix TestOzoneRpcClientAbstract.testListVolume. (#1194)
new 69bb0ee HDDS-3062. Fix TestOzoneRpcClientAbstract.testListVolume - addendum for checkstyle
new b3139e8 HDDS-3957. Fix mixed use of Longs.toByteArray and Ints.fromByteArray (#1199)
new f566187 HDDS-3798. Display version and setupTime of DN in recon web (#1136)
new 99e7b7a HDDS-3060. /retest github comment does not work (#1137)
new d97fa28 HDDS-3925. SCM Pipeline DB should directly use UUID bytes for key rather than rely on proto serialization for key. (#1197)
new 53f4084 HDDS-3885. Create Datanode home page (#1139)
new 880adec HDDS-2768. security/SecuringS3.md (#1188)
new 06e186b HDDS-2769. security/SecurityWithRanger.md (#1189)
new 2a66d32 HDDS-3798. Display more accurate timestamp in recon Web (#1201)
new adb15ba HDDS-3765. Fluentd writing to secure Ozone S3 API fails with 500 Error. (#1179)
new e65d196 HDDS-3967. Remove leftover debug setting (#1202)
new 0c8f2c8 HDDS-2767. security/SecuringTDE.md (#1184)
new 813a4c4 HDDS-3958. Intermittent failure in Recon acceptance test due to mixed stdout and stderr (#1200)
new c606a84 HDDS-3923. Display the safemode status on scm page (#1165)
new 82657a7 HDDS-3807. Propagate raft log disks info to SCM from datanode. (#1107)
new d4c6f7d HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK (#1164)
new 6b666bf HDDS-3612. Allow mounting bucket under other volume (#1104)
new 075423b HDDS-3964. Ratis config key mismatch (#1204)
new 6361f4a HDDS-3855. Add upgrade smoketest (#1142)
new 6cdfc7d HDDS-3965. SCM failed to start up for duplicated pipeline detected. (#1210)
new d92bcba HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (#1129)
new ba8966d HDDS-3955. Unable to list intermediate paths on keys created using S3G. (#1196)
new 8b346ed HDDS-3806. Support recognize aws v2 Authorization header. (#1098)
new de093ec HDDS-3987. Encrypted bucket creation failed with INVALID_REQUEST Encryption cannot be set for bucket links (#1221)
new a90b7bb HDDS-3982. Disable moveToTrash in o3fs and ofs temporarily (#1215)
new f58883b Update ratis to 1.0.0 (#1222)
new a86afab HDDS-3813. Upgrade Ratis third-party, too (#1229)
new d9775d4 HDDS-3986. Frequent failure in TestCommitWatcher#testReleaseBuffersOnException (#1220)
new 2a79c89 HDDS-3892. Datanode initialization is too slow when there are thousan… (#1147)
new 507bba5 HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread (#1185)
new 5ac4073 HDDS-2770. security/SecurityAcls.md (#1190)
The 39 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/comment-commands/retest.sh | 66 ++-
.github/workflows/post-commit.yml | 1 +
.gitignore | 2 +
LICENSE.txt | 4 +-
hadoop-hdds/common/src/main/conf/hadoop-env.sh | 3 +
.../hadoop/hdds/protocol/DatanodeDetails.java | 90 +++-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 24 +-
.../hadoop/hdds/ratis/conf/RatisClientConfig.java | 6 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 12 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +
.../common/src/main/resources/ozone-default.xml | 45 +-
hadoop-hdds/container-service/pom.xml | 31 ++
.../java/org/apache/hadoop/ozone/DNMXBean.java | 8 +-
...sDatanodeStopService.java => DNMXBeanImpl.java} | 15 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 35 +-
.../common/impl/StorageLocationReport.java | 11 +-
...TopNOrderedContainerDeletionChoosingPolicy.java | 2 +-
.../common/statemachine/DatanodeStateMachine.java | 25 +-
.../common/statemachine/StateContext.java | 34 +-
.../states/datanode/RunningDatanodeState.java | 14 +-
.../common/transport/server/XceiverServerSpi.java | 12 +-
.../server/ratis/ContainerStateMachine.java | 12 +-
.../transport/server/ratis/XceiverServerRatis.java | 66 ++-
.../container/common/utils/ContainerCache.java | 71 ++-
.../container/common/volume/MutableVolumeSet.java | 10 +-
.../container/keyvalue/KeyValueContainer.java | 5 -
.../container/keyvalue/KeyValueContainerCheck.java | 8 +-
.../container/keyvalue/KeyValueContainerData.java | 17 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 3 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 21 +-
.../main/resources/webapps/hddsDatanode/.gitkeep | 17 -
.../webapps/hddsDatanode/dn-overview.html | 8 +-
.../src/main/resources/webapps/hddsDatanode/dn.js} | 21 +-
.../resources/webapps/hddsDatanode}/index.html | 10 +-
.../main/resources/webapps/hddsDatanode}/main.html | 4 +-
.../container/common/TestBlockDeletingService.java | 7 +-
.../ozone/container/common/TestContainerCache.java | 52 +++
.../common/statemachine/TestStateContext.java | 30 ++
.../states/datanode/TestRunningDatanodeState.java | 90 ++++
.../keyvalue/TestKeyValueBlockIterator.java | 16 +-
.../keyvalue/TestKeyValueContainerCheck.java | 19 +-
.../container/ozoneimpl/TestContainerReader.java | 69 ++-
.../container/ozoneimpl/TestOzoneContainer.java | 51 +-
hadoop-hdds/docs/content/design/ofs.md | 4 +
hadoop-hdds/docs/content/design/trash.md | 7 +-
hadoop-hdds/docs/content/interface/OzoneFS.md | 3 +
...ecuityWithRanger.md => SecuityWithRanger.zh.md} | 18 +-
hadoop-hdds/docs/content/security/SecuringS3.md | 6 +-
hadoop-hdds/docs/content/security/SecuringS3.zh.md | 50 ++
hadoop-hdds/docs/content/security/SecuringTDE.md | 4 +-
.../docs/content/security/SecuringTDE.zh.md | 52 +++
hadoop-hdds/docs/content/security/SecurityAcls.md | 8 +-
.../docs/content/security/SecurityAcls.zh.md | 66 +++
hadoop-hdds/docs/pom.xml | 2 +-
.../themes/ozonedoc/layouts/partials/footer.html | 2 +-
.../themes/ozonedoc/static/js/jquery-3.4.1.min.js | 2 -
.../themes/ozonedoc/static/js/jquery-3.5.1.min.js | 2 +
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 7 +-
.../hdds/conf/DatanodeRatisServerConfig.java | 32 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 16 +-
.../org/apache/hadoop/hdds/utils/LevelDBStore.java | 1 +
.../hadoop/hdds/utils/LevelDBStoreIterator.java | 1 +
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 25 +-
.../hadoop/hdds/utils/db/RDBStoreIterator.java | 52 ++-
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 2 +-
.../apache/hadoop/hdds/utils/db/TableIterator.java | 8 +
.../apache/hadoop/hdds/utils/db/TypedTable.java | 5 +
.../resources/webapps/static/jquery-3.4.1.min.js | 2 -
.../resources/webapps/static/jquery-3.5.1.min.js | 2 +
.../hadoop/hdds/utils/TestMetadataStore.java | 48 +-
.../hadoop/hdds/utils/TestRocksDBStoreMBean.java | 3 -
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 226 +++++++++
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 61 +++
.../interface-client/src/main/proto/hdds.proto | 2 +
.../interface-client/src/main/proto/proto.lock | 12 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 6 +
hadoop-hdds/pom.xml | 2 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 2 +-
.../hdds/scm/container/ContainerReportHandler.java | 9 +-
.../IncrementalContainerReportHandler.java | 13 +-
.../hdds/scm/container/ReplicationManager.java | 12 +-
.../hadoop/hdds/scm/metadata/PipelineIDCodec.java | 38 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 63 +--
.../hdds/scm/pipeline/SCMPipelineManager.java | 60 ++-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 2 +
.../hdds/scm/server/StorageContainerManager.java | 10 +
.../src/main/resources/webapps/scm/index.html | 2 +-
.../main/resources/webapps/scm/scm-overview.html | 11 +
.../TestIncrementalContainerReportHandler.java | 36 +-
.../hdds/scm/metadata/TestPipelineIDCodec.java | 144 ++++++
.../hdds/scm/node/TestNodeReportHandler.java | 5 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 2 +
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 195 ++++++++
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 180 +++++++
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 5 +-
.../org/apache/hadoop/ozone/client/BucketArgs.java | 33 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 23 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 11 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 5 -
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 7 +
.../hadoop/ozone/om/exceptions/OMException.java | 4 +-
.../ozone/om/helpers/BucketEncryptionKeyInfo.java | 4 +
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 142 ++++--
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 18 +
.../{BooleanBiFunction.java => ExitManager.java} | 20 +-
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 22 +-
.../dist/dev-support/bin/dist-layout-stitching | 1 +
hadoop-ozone/dist/pom.xml | 2 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 18 +-
.../dist/src/main/compose/ozonesecure/test.sh | 15 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 39 +-
.../src/main/compose/{ozone-csi => upgrade}/.env | 4 +-
.../docker-image/docker-krb5 => upgrade}/README.md | 27 +-
.../compose/{ozone => upgrade}/docker-compose.yaml | 107 +++--
.../main/compose/{ozone => upgrade}/docker-config | 2 +-
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 70 +++
.../{ozonesecure => upgrade/versions}/README.md | 9 +-
.../stopOM.sh => upgrade/versions/ozone-0.5.0.sh} | 4 +-
.../stopOM.sh => upgrade/versions/ozone-0.6.0.sh} | 4 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 2 +-
.../dist/src/main/smoketest/basic/links.robot | 152 ++++++
.../dist/src/main/smoketest/commonlib.robot | 34 +-
.../smoketest/{commonlib.robot => lib/os.robot} | 36 +-
.../{s3/bucketlist.robot => lib/os_tests.robot} | 33 +-
.../dist/src/main/smoketest/ozone-lib/shell.robot | 53 +++
.../src/main/smoketest/ozone-lib/shell_tests.robot | 58 +++
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 2 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 16 +-
.../dist/src/main/smoketest/recon/recon-api.robot | 16 +-
.../smoketest/{lib/fs_tests.robot => robot.robot} | 48 +-
.../src/main/smoketest/s3/MultipartUpload.robot | 7 +-
.../dist/src/main/smoketest/s3/bucketdelete.robot | 12 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 33 +-
.../bucket-encryption.robot} | 26 +-
.../main/smoketest/security/ozone-secure-s3.robot | 15 +
.../ozone-csi/test.sh => shell/upgrade/0.6.0.sh} | 18 +-
.../upgrade/0.6.0/01-migrate-scm-db.sh} | 19 +-
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 132 ++++++
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 38 ++
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 37 ++
.../hadoop/fs/ozone/contract/OzoneContract.java | 17 +-
.../ozone/contract/rooted/RootedOzoneContract.java | 17 +-
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 2 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 16 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 28 +-
...estBlockOutputStreamWithFailuresFlushDelay.java | 28 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 41 +-
.../TestContainerStateMachineFailureOnRead.java | 27 +-
.../rpc/TestContainerStateMachineFailures.java | 27 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 28 +-
.../client/rpc/TestFailureHandlingByClient.java | 28 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 28 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 28 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 5 -
.../client/rpc/TestValidateBCSIDOnRestart.java | 28 +-
.../ozone/client/rpc/TestWatchForCommit.java | 28 +-
.../hadoop/ozone/freon/TestDataValidate.java | 17 +-
.../ozone/freon/TestFreonWithDatanodeRestart.java | 17 +-
.../ozone/freon/TestFreonWithPipelineDestroy.java | 17 +-
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 17 +-
.../ozone/om/TestContainerReportWithKeys.java | 10 +
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 215 +++++++--
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 100 ++--
.../org/apache/hadoop/ozone/om/TestOmSQLCli.java | 21 -
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 3 -
.../snapshot/TestOzoneManagerSnapshotProvider.java | 26 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 2 +
.../src/test/resources/log4j.properties | 3 -
.../src/main/proto/OmClientProtocol.proto | 4 +
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 112 +++--
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 164 ++++---
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 23 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 518 ++++++++++++++-------
.../org/apache/hadoop/ozone/om/ResolvedBucket.java | 111 +++++
.../hadoop/ozone/om/ratis/OMTransactionInfo.java | 21 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 6 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 37 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 76 +++
.../om/request/bucket/OMBucketCreateRequest.java | 14 +
.../om/request/file/OMDirectoryCreateRequest.java | 4 +
.../ozone/om/request/file/OMFileCreateRequest.java | 6 +-
.../ozone/om/request/file/OMFileRequest.java | 2 +-
.../om/request/key/OMAllocateBlockRequest.java | 15 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 31 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 76 ++-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 19 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 36 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 11 +
.../ozone/om/request/key/OMKeysDeleteRequest.java | 48 +-
.../om/request/key/OMTrashRecoverRequest.java | 7 +
.../S3InitiateMultipartUploadRequest.java | 47 +-
.../multipart/S3MultipartUploadAbortRequest.java | 30 +-
.../S3MultipartUploadCommitPartRequest.java | 51 +-
.../S3MultipartUploadCompleteRequest.java | 76 +--
.../om/snapshot/OzoneManagerSnapshotProvider.java | 10 +-
.../main/resources/webapps/ozoneManager/index.html | 2 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 2 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 2 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 2 +-
.../ozone/om/request/TestNormalizePaths.java | 109 +++++
.../request/file/TestOMDirectoryCreateRequest.java | 4 +
.../om/request/key/TestOMKeyCreateRequest.java | 182 +++++++-
.../ozone/om/request/key/TestOMKeyRequest.java | 8 +
.../TestS3InitiateMultipartUploadRequest.java | 2 +-
.../s3/multipart/TestS3MultipartRequest.java | 15 +-
.../TestS3MultipartUploadCommitPartRequest.java | 2 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 33 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 29 ++
hadoop-ozone/pom.xml | 2 +-
.../hadoop/ozone/recon/ReconServerConfigKeys.java | 7 -
.../hadoop/ozone/recon/api/NodeEndpoint.java | 2 +
.../ozone/recon/api/types/DatanodeMetadata.java | 28 ++
.../ReconIncrementalContainerReportHandler.java | 12 +-
.../webapps/recon/ozone-recon-web/api/db.json | 48 +-
.../components/autoReloadPanel/autoReloadPanel.tsx | 4 +-
.../src/components/multiSelect/multiSelect.tsx | 5 +-
.../src/views/datanodes/datanodes.less | 14 +
.../src/views/datanodes/datanodes.tsx | 102 +++-
.../src/views/pipelines/pipelines.tsx | 2 +-
...TestReconIncrementalContainerReportHandler.java | 30 +-
...reProcessor.java => AWSSignatureProcessor.java} | 43 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 8 +-
.../ozone/s3/header/AuthorizationHeaderV2.java | 2 +-
.../ozone/s3/header/AuthorizationHeaderV4.java | 4 +-
.../s3gateway/src/main/resources/browser.html | 2 +-
.../src/main/resources/webapps/static/index.html | 2 +-
...ocessor.java => TestAWSSignatureProcessor.java} | 44 +-
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 2 +-
.../ozone/s3/header/TestAuthorizationHeaderV4.java | 2 +-
.../hadoop/ozone/shell/bucket/BucketCommands.java | 1 +
...teBucketHandler.java => LinkBucketHandler.java} | 57 +--
pom.xml | 4 +-
233 files changed, 5560 insertions(+), 1600 deletions(-)
copy hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMXBean.java => hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBean.java (84%)
copy hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/{HddsDatanodeStopService.java => DNMXBeanImpl.java} (72%)
delete mode 100644 hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
copy hadoop-ozone/common/dev-support/findbugsExcludeFile.xml => hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html (88%)
copy hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/cli/package-info.java => container-service/src/main/resources/webapps/hddsDatanode/dn.js} (60%)
copy hadoop-hdds/{server-scm/src/main/resources/webapps/scm => container-service/src/main/resources/webapps/hddsDatanode}/index.html (92%)
copy hadoop-hdds/{server-scm/src/main/resources/webapps/scm => container-service/src/main/resources/webapps/hddsDatanode}/main.html (95%)
create mode 100644 hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/datanode/TestRunningDatanodeState.java
copy hadoop-hdds/docs/content/security/{SecuityWithRanger.md => SecuityWithRanger.zh.md} (54%)
create mode 100644 hadoop-hdds/docs/content/security/SecuringS3.zh.md
create mode 100644 hadoop-hdds/docs/content/security/SecuringTDE.zh.md
create mode 100644 hadoop-hdds/docs/content/security/SecurityAcls.zh.md
delete mode 100644 hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js
create mode 100644 hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js
delete mode 100644 hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.4.1.min.js
create mode 100644 hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.5.1.min.js
create mode 100644 hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/metadata/TestPipelineIDCodec.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/{BooleanBiFunction.java => ExitManager.java} (69%)
copy hadoop-ozone/dist/src/main/compose/{ozone-csi => upgrade}/.env (89%)
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 => upgrade}/README.md (50%)
copy hadoop-ozone/dist/src/main/compose/{ozone => upgrade}/docker-compose.yaml (59%)
copy hadoop-ozone/dist/src/main/compose/{ozone => upgrade}/docker-config (96%)
create mode 100644 hadoop-ozone/dist/src/main/compose/upgrade/test.sh
copy hadoop-ozone/dist/src/main/compose/{ozonesecure => upgrade/versions}/README.md (70%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/stopOM.sh => upgrade/versions/ozone-0.5.0.sh} (94%)
mode change 100755 => 100644
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/stopOM.sh => upgrade/versions/ozone-0.6.0.sh} (94%)
mode change 100755 => 100644
create mode 100644 hadoop-ozone/dist/src/main/smoketest/basic/links.robot
copy hadoop-ozone/dist/src/main/smoketest/{commonlib.robot => lib/os.robot} (57%)
copy hadoop-ozone/dist/src/main/smoketest/{s3/bucketlist.robot => lib/os_tests.robot} (58%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell_tests.robot
copy hadoop-ozone/dist/src/main/smoketest/{lib/fs_tests.robot => robot.robot} (61%)
copy hadoop-ozone/dist/src/main/smoketest/{admincli/pipeline.robot => security/bucket-encryption.robot} (60%)
copy hadoop-ozone/dist/src/{main/compose/ozone-csi/test.sh => shell/upgrade/0.6.0.sh} (71%)
copy hadoop-ozone/dist/src/{main/compose/ozone-csi/test.sh => shell/upgrade/0.6.0/01-migrate-scm-db.sh} (67%)
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSWithObjectStoreCreate.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
rename hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/{AWSV4SignatureProcessor.java => AWSSignatureProcessor.java} (91%)
rename hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/{TestAWSV4SignatureProcessor.java => TestAWSSignatureProcessor.java} (69%)
copy hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/{CreateBucketHandler.java => LinkBucketHandler.java} (58%)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 13/39: HDDS-3925. SCM Pipeline DB should directly
use UUID bytes for key rather than rely on proto serialization for key.
(#1197)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit d97fa2823ea2de380a68dc83e050641cde4a258a
Author: Istvan Fajth <pi...@cloudera.com>
AuthorDate: Tue Jul 14 23:30:45 2020 +0200
HDDS-3925. SCM Pipeline DB should directly use UUID bytes for key rather than rely on proto serialization for key. (#1197)
(cherry picked from commit 0a1cce5b98c22f72b31ca13a8ef2734f86b0bec4)
---
.../hadoop/hdds/utils/db/RDBStoreIterator.java | 16 ++
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 2 +-
.../apache/hadoop/hdds/utils/db/TableIterator.java | 8 +
.../apache/hadoop/hdds/utils/db/TypedTable.java | 5 +
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 224 +++++++++++++++++++++
.../hadoop/hdds/scm/metadata/PipelineIDCodec.java | 38 +++-
.../hdds/scm/pipeline/SCMPipelineManager.java | 54 ++++-
.../hdds/scm/metadata/TestPipelineIDCodec.java | 144 +++++++++++++
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 115 +++++++++++
9 files changed, 602 insertions(+), 4 deletions(-)
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
index 784738b..5902486 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
@@ -32,12 +32,18 @@ public class RDBStoreIterator
implements TableIterator<byte[], ByteArrayKeyValue> {
private RocksIterator rocksDBIterator;
+ private RDBTable rocksDBTable;
public RDBStoreIterator(RocksIterator iterator) {
this.rocksDBIterator = iterator;
rocksDBIterator.seekToFirst();
}
+ public RDBStoreIterator(RocksIterator iterator, RDBTable table) {
+ this(iterator);
+ this.rocksDBTable = table;
+ }
+
@Override
public void forEachRemaining(
Consumer<? super ByteArrayKeyValue> action) {
@@ -101,6 +107,16 @@ public class RDBStoreIterator
}
@Override
+ public void removeFromDB() throws IOException {
+ if (rocksDBTable == null) {
+ throw new UnsupportedOperationException("remove");
+ }
+ if (rocksDBIterator.isValid()) {
+ rocksDBTable.delete(rocksDBIterator.key());
+ }
+ }
+
+ @Override
public void close() throws IOException {
rocksDBIterator.close();
}
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java
index 2e390e2..4dbb59a 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBTable.java
@@ -206,7 +206,7 @@ class RDBTable implements Table<byte[], byte[]> {
public TableIterator<byte[], ByteArrayKeyValue> iterator() {
ReadOptions readOptions = new ReadOptions();
readOptions.setFillCache(false);
- return new RDBStoreIterator(db.newIterator(handle, readOptions));
+ return new RDBStoreIterator(db.newIterator(handle, readOptions), this);
}
@Override
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java
index a684157..c9bc045 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableIterator.java
@@ -60,4 +60,12 @@ public interface TableIterator<KEY, T> extends Iterator<T>, Closeable {
*/
T value();
+ /**
+ * Remove the actual value of the iterator from the database table on
+ * which the iterator is working on.
+ *
+ * @throws IOException when there is an error occured during deletion.
+ */
+ void removeFromDB() throws IOException;
+
}
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java
index 86d23af..1451946 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java
@@ -420,5 +420,10 @@ public class TypedTable<KEY, VALUE> implements Table<KEY, VALUE> {
return new TypedKeyValue(rawIterator.next(), keyType,
valueType);
}
+
+ @Override
+ public void removeFromDB() throws IOException {
+ rawIterator.removeFromDB();
+ }
}
}
diff --git a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
new file mode 100644
index 0000000..6e85977
--- /dev/null
+++ b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.hadoop.hdds.utils.db;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.InOrder;
+import org.rocksdb.RocksIterator;
+
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * This test prescribe expected behaviour from the RDBStoreIterator which wraps
+ * RocksDB's own iterator. Ozone internally in TypedTableIterator uses, the
+ * RDBStoreIterator to provide iteration over table elements in a typed manner.
+ * The tests are to ensure we access RocksDB via the iterator properly.
+ */
+public class TestRDBStoreIterator {
+
+ private RocksIterator rocksDBIteratorMock;
+ private RDBTable rocksTableMock;
+
+ @Before
+ public void setup() {
+ rocksDBIteratorMock = mock(RocksIterator.class);
+ rocksTableMock = mock(RDBTable.class);
+ }
+
+ @Test
+ public void testForeachRemainingCallsConsumerWithAllElements() {
+ when(rocksDBIteratorMock.isValid())
+ .thenReturn(true, true, true, true, true, true, false);
+ when(rocksDBIteratorMock.key())
+ .thenReturn(new byte[]{0x00}, new byte[]{0x01}, new byte[]{0x02})
+ .thenThrow(new NoSuchElementException());
+ when(rocksDBIteratorMock.value())
+ .thenReturn(new byte[]{0x7f}, new byte[]{0x7e}, new byte[]{0x7d})
+ .thenThrow(new NoSuchElementException());
+
+
+ Consumer<ByteArrayKeyValue> consumerStub = mock(Consumer.class);
+
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ iter.forEachRemaining(consumerStub);
+
+ ArgumentCaptor<ByteArrayKeyValue> capture =
+ ArgumentCaptor.forClass(ByteArrayKeyValue.class);
+ verify(consumerStub, times(3)).accept(capture.capture());
+ assertArrayEquals(
+ new byte[]{0x00}, capture.getAllValues().get(0).getKey());
+ assertArrayEquals(
+ new byte[]{0x7f}, capture.getAllValues().get(0).getValue());
+ assertArrayEquals(
+ new byte[]{0x01}, capture.getAllValues().get(1).getKey());
+ assertArrayEquals(
+ new byte[]{0x7e}, capture.getAllValues().get(1).getValue());
+ assertArrayEquals(
+ new byte[]{0x02}, capture.getAllValues().get(2).getKey());
+ assertArrayEquals(
+ new byte[]{0x7d}, capture.getAllValues().get(2).getValue());
+ }
+
+ @Test
+ public void testHasNextDependsOnIsvalid(){
+ when(rocksDBIteratorMock.isValid()).thenReturn(true, false);
+
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+
+ assertTrue(iter.hasNext());
+ assertFalse(iter.hasNext());
+ }
+
+ @Test
+ public void testNextCallsIsValidThenGetsTheValueAndStepsToNext() {
+ when(rocksDBIteratorMock.isValid()).thenReturn(true);
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+
+ InOrder verifier = inOrder(rocksDBIteratorMock);
+
+ iter.next();
+
+ verifier.verify(rocksDBIteratorMock).isValid();
+ verifier.verify(rocksDBIteratorMock).key();
+ verifier.verify(rocksDBIteratorMock).value();
+ verifier.verify(rocksDBIteratorMock).next();
+ }
+
+ @Test
+ public void testConstructorSeeksToFirstElement() {
+ new RDBStoreIterator(rocksDBIteratorMock);
+
+ verify(rocksDBIteratorMock, times(1)).seekToFirst();
+ }
+
+ @Test
+ public void testSeekToFirstSeeks() {
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+
+ iter.seekToFirst();
+
+ verify(rocksDBIteratorMock, times(2)).seekToFirst();
+ }
+
+ @Test
+ public void testSeekToLastSeeks() {
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+
+ iter.seekToLast();
+
+ verify(rocksDBIteratorMock, times(1)).seekToLast();
+ }
+
+ @Test
+ public void testSeekReturnsTheActualKey() {
+ when(rocksDBIteratorMock.isValid()).thenReturn(true);
+ when(rocksDBIteratorMock.key()).thenReturn(new byte[]{0x00});
+ when(rocksDBIteratorMock.value()).thenReturn(new byte[]{0x7f});
+
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ ByteArrayKeyValue val = iter.seek(new byte[]{0x55});
+
+ InOrder verifier = inOrder(rocksDBIteratorMock);
+
+ verify(rocksDBIteratorMock, times(1)).seekToFirst(); //at construct time
+ verify(rocksDBIteratorMock, never()).seekToLast();
+ verifier.verify(rocksDBIteratorMock, times(1)).seek(any(byte[].class));
+ verifier.verify(rocksDBIteratorMock, times(1)).isValid();
+ verifier.verify(rocksDBIteratorMock, times(1)).key();
+ verifier.verify(rocksDBIteratorMock, times(1)).value();
+ assertArrayEquals(new byte[]{0x00}, val.getKey());
+ assertArrayEquals(new byte[]{0x7f}, val.getValue());
+ }
+
+ @Test
+ public void testGettingTheKeyIfIteratorIsValid() {
+ when(rocksDBIteratorMock.isValid()).thenReturn(true);
+ when(rocksDBIteratorMock.key()).thenReturn(new byte[]{0x00});
+
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ byte[] key = iter.key();
+
+ InOrder verifier = inOrder(rocksDBIteratorMock);
+
+ verifier.verify(rocksDBIteratorMock, times(1)).isValid();
+ verifier.verify(rocksDBIteratorMock, times(1)).key();
+ assertArrayEquals(new byte[]{0x00}, key);
+ }
+
+ @Test
+ public void testGettingTheValueIfIteratorIsValid() {
+ when(rocksDBIteratorMock.isValid()).thenReturn(true);
+ when(rocksDBIteratorMock.key()).thenReturn(new byte[]{0x00});
+ when(rocksDBIteratorMock.value()).thenReturn(new byte[]{0x7f});
+
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ ByteArrayKeyValue val = iter.value();
+
+ InOrder verifier = inOrder(rocksDBIteratorMock);
+
+ verifier.verify(rocksDBIteratorMock, times(1)).isValid();
+ verifier.verify(rocksDBIteratorMock, times(1)).key();
+ assertArrayEquals(new byte[]{0x00}, val.getKey());
+ assertArrayEquals(new byte[]{0x7f}, val.getValue());
+ }
+
+ @Test
+ public void testRemovingFromDBActuallyDeletesFromTable() throws Exception {
+ byte[] testKey = new byte[]{0x00};
+ when(rocksDBIteratorMock.isValid()).thenReturn(true);
+ when(rocksDBIteratorMock.key()).thenReturn(testKey);
+
+ RDBStoreIterator iter =
+ new RDBStoreIterator(rocksDBIteratorMock, rocksTableMock);
+ iter.removeFromDB();
+
+ InOrder verifier = inOrder(rocksDBIteratorMock, rocksTableMock);
+
+ verifier.verify(rocksDBIteratorMock, times(1)).isValid();
+ verifier.verify(rocksTableMock, times(1)).delete(testKey);
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testRemoveFromDBWithoutDBTableSet() throws Exception {
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ iter.removeFromDB();
+ }
+
+ @Test
+ public void testCloseCloses() throws Exception {
+ RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
+ iter.close();
+
+ verify(rocksDBIteratorMock, times(1)).close();
+ }
+}
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineIDCodec.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineIDCodec.java
index d661e34..e73539f 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineIDCodec.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/PipelineIDCodec.java
@@ -19,6 +19,9 @@
package org.apache.hadoop.hdds.scm.metadata;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.UUID;
import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
import org.apache.hadoop.hdds.utils.db.Codec;
@@ -30,12 +33,43 @@ public class PipelineIDCodec implements Codec<PipelineID> {
@Override
public byte[] toPersistedFormat(PipelineID object) throws IOException {
- return object.getProtobuf().toByteArray();
+ byte[] bytes = new byte[16];
+ System.arraycopy(
+ asByteArray(object.getId().getMostSignificantBits()), 0, bytes, 0, 8);
+ System.arraycopy(
+ asByteArray(object.getId().getLeastSignificantBits()), 0, bytes, 8, 8);
+ return bytes;
+ }
+
+ private byte[] asByteArray(long bits) {
+ ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.putLong(bits);
+ return buffer.array();
}
@Override
public PipelineID fromPersistedFormat(byte[] rawData) throws IOException {
- return null;
+ long mostSiginificantBits = toLong(rawData, 0);
+ long leastSignificantBits = toLong(rawData, 8);
+
+ UUID id = new UUID(mostSiginificantBits, leastSignificantBits);
+ return PipelineID.valueOf(id);
+ }
+
+ private long toLong(byte[] arr, int startIdx) throws IOException {
+ if (arr.length < startIdx + 8) {
+ throw new IOException("Key conversion error.",
+ new ArrayIndexOutOfBoundsException(
+ "Key does not have the least expected amount of bytes,"
+ + "and does not contain a UUID. Key: "
+ + Arrays.toString(arr)
+ )
+ );
+ }
+ ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
+ buffer.put(arr, startIdx, 8);
+ buffer.flip();
+ return buffer.getLong();
}
@Override
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
index e8223ca..fda9371 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
@@ -161,12 +161,64 @@ public class SCMPipelineManager implements PipelineManager {
TableIterator<PipelineID, ? extends KeyValue<PipelineID, Pipeline>>
iterator = pipelineStore.iterator();
while (iterator.hasNext()) {
- Pipeline pipeline = iterator.next().getValue();
+ Pipeline pipeline = nextPipelineFromIterator(iterator);
stateManager.addPipeline(pipeline);
nodeManager.addPipeline(pipeline);
}
}
+ private Pipeline nextPipelineFromIterator(
+ TableIterator<PipelineID, ? extends KeyValue<PipelineID, Pipeline>> it
+ ) throws IOException {
+ KeyValue<PipelineID, Pipeline> actual = it.next();
+ Pipeline pipeline = actual.getValue();
+ PipelineID pipelineID = actual.getKey();
+ checkKeyAndReplaceIfObsolete(it, pipeline, pipelineID);
+ return pipeline;
+ }
+
+ /**
+ * This method is part of the change that happens in HDDS-3925, and we can
+ * and should remove this on later on.
+ * The purpose of the change is to get rid of protobuf serialization in the
+ * SCM database Pipeline table keys. The keys are not used anywhere, and the
+ * PipelineID that is used as a key is in the value as well, so we can detect
+ * a change in the key translation to byte[] and if we have the old format
+ * we refresh the table contents during SCM startup.
+ *
+ * If this fails in the remove, then there is an IOException coming from
+ * RocksDB itself, in this case in memory structures will still be fine and
+ * SCM should be operational, however we will attempt to replace the old key
+ * at next startup. In this case removing of the pipeline will leave the
+ * pipeline in RocksDB, and during next startup we will attempt to delete it
+ * again. This does not affect any runtime operations.
+ * If a Pipeline should have been deleted but remained in RocksDB, then at
+ * next startup it will be replaced and added with the new key, then SCM will
+ * detect that it is an invalid Pipeline and successfully delete it with the
+ * new key.
+ * For further info check the JIRA.
+ *
+ * @param it the iterator used to iterate the Pipeline table
+ * @param pipeline the pipeline read already from the iterator
+ * @param pipelineID the pipeline ID read from the raw data via the iterator
+ */
+ private void checkKeyAndReplaceIfObsolete(
+ TableIterator<PipelineID, ? extends KeyValue<PipelineID, Pipeline>> it,
+ Pipeline pipeline,
+ PipelineID pipelineID
+ ) {
+ if (!pipelineID.equals(pipeline.getId())) {
+ try {
+ it.removeFromDB();
+ pipelineStore.put(pipeline.getId(), pipeline);
+ } catch (IOException e) {
+ LOG.info("Pipeline table in RocksDB has an old key format, and "
+ + "removing the pipeline with the old key was unsuccessful."
+ + "Pipeline: {}", pipeline);
+ }
+ }
+ }
+
private void recordMetricsForPipeline(Pipeline pipeline) {
metrics.incNumPipelineAllocated();
if (pipeline.isOpen()) {
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/metadata/TestPipelineIDCodec.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/metadata/TestPipelineIDCodec.java
new file mode 100644
index 0000000..5543be5
--- /dev/null
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/metadata/TestPipelineIDCodec.java
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.metadata;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.junit.Test;
+
+import java.util.UUID;
+
+/**
+ * Testing serialization of PipelineID objects to/from RocksDB.
+ */
+public class TestPipelineIDCodec {
+
+ @Test
+ public void testPersistingZeroAsUUID() throws Exception {
+ long leastSigBits = 0x0000_0000_0000_0000L;
+ long mostSigBits = 0x0000_0000_0000_0000L;
+ byte[] expected = new byte[] {
+ b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00),
+ b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00), b(0x00)
+ };
+
+ checkPersisting(leastSigBits, mostSigBits, expected);
+ }
+
+ @Test
+ public void testPersistingFFAsUUID() throws Exception {
+ long leastSigBits = 0xFFFF_FFFF_FFFF_FFFFL;
+ long mostSigBits = 0xFFFF_FFFF_FFFF_FFFFL;
+ byte[] expected = new byte[] {
+ b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF),
+ b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF), b(0xFF)
+ };
+
+ checkPersisting(leastSigBits, mostSigBits, expected);
+ }
+
+ @Test
+ public void testPersistingARandomUUID() throws Exception {
+ for (int i=0; i<100; i++) {
+ UUID uuid = UUID.randomUUID();
+
+ long mask = 0x0000_0000_0000_00FFL;
+
+ byte[] expected = new byte[] {
+ b(((int) (uuid.getMostSignificantBits() >> 56 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 48 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 40 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 32 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 24 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 16 & mask))),
+ b(((int) (uuid.getMostSignificantBits() >> 8 & mask))),
+ b(((int) (uuid.getMostSignificantBits() & mask))),
+
+ b(((int) (uuid.getLeastSignificantBits() >> 56 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 48 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 40 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 32 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 24 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 16 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() >> 8 & mask))),
+ b(((int) (uuid.getLeastSignificantBits() & mask))),
+ };
+
+ checkPersisting(
+ uuid.getMostSignificantBits(),
+ uuid.getLeastSignificantBits(),
+ expected
+ );
+ }
+ }
+
+ @Test
+ public void testConvertAndReadBackZeroAsUUID() throws Exception {
+ long mostSigBits = 0x0000_0000_0000_0000L;
+ long leastSigBits = 0x0000_0000_0000_0000L;
+ UUID uuid = new UUID(mostSigBits, leastSigBits);
+ PipelineID pid = PipelineID.valueOf(uuid);
+
+ byte[] encoded = new PipelineIDCodec().toPersistedFormat(pid);
+ PipelineID decoded = new PipelineIDCodec().fromPersistedFormat(encoded);
+
+ assertEquals(pid, decoded);
+ }
+
+ @Test
+ public void testConvertAndReadBackFFAsUUID() throws Exception {
+ long mostSigBits = 0xFFFF_FFFF_FFFF_FFFFL;
+ long leastSigBits = 0xFFFF_FFFF_FFFF_FFFFL;
+ UUID uuid = new UUID(mostSigBits, leastSigBits);
+ PipelineID pid = PipelineID.valueOf(uuid);
+
+ byte[] encoded = new PipelineIDCodec().toPersistedFormat(pid);
+ PipelineID decoded = new PipelineIDCodec().fromPersistedFormat(encoded);
+
+ assertEquals(pid, decoded);
+ }
+
+ @Test
+ public void testConvertAndReadBackRandomUUID() throws Exception {
+ UUID uuid = UUID.randomUUID();
+ PipelineID pid = PipelineID.valueOf(uuid);
+
+ byte[] encoded = new PipelineIDCodec().toPersistedFormat(pid);
+ PipelineID decoded = new PipelineIDCodec().fromPersistedFormat(encoded);
+
+ assertEquals(pid, decoded);
+ }
+
+ private void checkPersisting(
+ long mostSigBits, long leastSigBits, byte[] expected
+ ) throws Exception {
+ UUID uuid = new UUID(mostSigBits, leastSigBits);
+ PipelineID pid = PipelineID.valueOf(uuid);
+
+ byte[] encoded = new PipelineIDCodec().toPersistedFormat(pid);
+
+ assertArrayEquals(expected, encoded);
+ }
+
+ private byte b(int i) {
+ return (byte) (i & 0x0000_00FF);
+ }
+}
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
index 7c2f17e..fc8f61a 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
@@ -21,8 +21,10 @@ package org.apache.hadoop.hdds.scm.pipeline;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
+import java.util.NoSuchElementException;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
@@ -37,11 +39,15 @@ import org.apache.hadoop.hdds.scm.TestUtils;
import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.MockNodeManager;
import org.apache.hadoop.hdds.scm.exceptions.SCMException;
+import org.apache.hadoop.hdds.scm.metadata.PipelineIDCodec;
import org.apache.hadoop.hdds.scm.metadata.SCMMetadataStore;
import org.apache.hadoop.hdds.scm.metadata.SCMMetadataStoreImpl;
import org.apache.hadoop.hdds.scm.safemode.SCMSafeModeManager;
import org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.PipelineReportFromDatanode;
import org.apache.hadoop.hdds.server.events.EventQueue;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.Table.KeyValue;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
import org.apache.hadoop.metrics2.MetricsRecordBuilder;
import org.apache.hadoop.test.GenericTestUtils;
@@ -56,7 +62,14 @@ import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
import org.junit.Before;
import org.junit.Test;
+import org.mockito.InOrder;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
/**
* Test cases to verify PipelineManager.
@@ -539,6 +552,108 @@ public class TestSCMPipelineManager {
pipelineManager.close();
}
+ /**
+ * This test was created for HDDS-3925 to check whether the db handling is
+ * proper at the SCMPipelineManager level. We should remove this test
+ * when we remove the key swap from the SCMPipelineManager code.
+ *
+ * The test emulates internally the values that the iterator will provide
+ * back to the check-fix code path. The iterator internally deserialize the
+ * key stored in RocksDB using the PipelineIDCodec. The older version of the
+ * codec serialized the PipelineIDs by taking the byte[] representation of
+ * the protobuf representation of the PipelineID, and deserialization was not
+ * implemented.
+ *
+ * In order to be able to check and fix the change, the deserialization was
+ * introduced, and deserialisation of the old protobuf byte representation
+ * with the new deserialization logic of the keys are
+ * checked against the PipelineID serialized in the value as well via
+ * protobuf.
+ * The DB is storing the keys now based on a byte[] serialized from the UUID
+ * inside the PipelineID.
+ * For this we emulate the getKey of the KeyValue returned by the
+ * iterator to return a PipelineID that is deserialized from the byte[]
+ * representation of the protobuf representation of the PipelineID in the
+ * test, as that would be the value we get from the iterator when iterating
+ * through a table with the old key format.
+ *
+ * @throws Exception when something goes wrong
+ */
+ @Test
+ public void testPipelineDBKeyFormatChange() throws Exception {
+ Pipeline p1 = pipelineStub();
+ Pipeline p2 = pipelineStub();
+ Pipeline p3 = pipelineStub();
+
+ TableIterator<PipelineID, KeyValue<PipelineID, Pipeline>> iteratorMock =
+ mock(TableIterator.class);
+
+ KeyValue<PipelineID, Pipeline> kv1 =
+ mockKeyValueToProvideOldKeyFormat(p1);
+ KeyValue<PipelineID, Pipeline> kv2 =
+ mockKeyValueToProvideNormalFormat(p2);
+ KeyValue<PipelineID, Pipeline> kv3 =
+ mockKeyValueToProvideOldKeyFormat(p3);
+
+ when(iteratorMock.next())
+ .thenReturn(kv1, kv2, kv3)
+ .thenThrow(new NoSuchElementException());
+ when(iteratorMock.hasNext())
+ .thenReturn(true, true, true, false);
+
+ Table<PipelineID, Pipeline> pipelineStore = mock(Table.class);
+ doReturn(iteratorMock).when(pipelineStore).iterator();
+ when(pipelineStore.isEmpty()).thenReturn(false);
+
+ InOrder inorderVerifier = inOrder(pipelineStore, iteratorMock);
+
+ new SCMPipelineManager(conf, nodeManager, pipelineStore, new EventQueue());
+
+ inorderVerifier.verify(iteratorMock).removeFromDB();
+ inorderVerifier.verify(pipelineStore).put(p1.getId(), p1);
+ inorderVerifier.verify(iteratorMock).removeFromDB();
+ inorderVerifier.verify(pipelineStore).put(p3.getId(), p3);
+
+ verify(pipelineStore, never()).put(p2.getId(), p2);
+ }
+
+ private Pipeline pipelineStub() {
+ return Pipeline.newBuilder()
+ .setId(PipelineID.randomId())
+ .setType(HddsProtos.ReplicationType.RATIS)
+ .setFactor(HddsProtos.ReplicationFactor.ONE)
+ .setState(Pipeline.PipelineState.OPEN)
+ .setNodes(
+ Arrays.asList(
+ nodeManager.getNodes(HddsProtos.NodeState.HEALTHY).get(0)
+ )
+ )
+ .setNodesInOrder(Arrays.asList(0))
+ .build();
+ }
+
+ private KeyValue<PipelineID, Pipeline>
+ mockKeyValueToProvideOldKeyFormat(Pipeline pipeline)
+ throws IOException {
+ KeyValue<PipelineID, Pipeline> kv = mock(KeyValue.class);
+ when(kv.getValue()).thenReturn(pipeline);
+ when(kv.getKey())
+ .thenReturn(
+ new PipelineIDCodec().fromPersistedFormat(
+ pipeline.getId().getProtobuf().toByteArray()
+ ));
+ return kv;
+ }
+
+ private KeyValue<PipelineID, Pipeline>
+ mockKeyValueToProvideNormalFormat(Pipeline pipeline)
+ throws IOException {
+ KeyValue<PipelineID, Pipeline> kv = mock(KeyValue.class);
+ when(kv.getValue()).thenReturn(pipeline);
+ when(kv.getKey()).thenReturn(pipeline.getId());
+ return kv;
+ }
+
private void sendPipelineReport(DatanodeDetails dn,
Pipeline pipeline, PipelineReportHandler pipelineReportHandler,
boolean isLeader, EventQueue eventQueue) {
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 07/39: HDDS-3789. Fix
TestOzoneRpcClientAbstract#testDeletedKeyForGDPR. (#1198)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 3a75437d4fc4cfa263d96bf2ca4b6d513e2e6759
Author: Lokesh Jain <lj...@apache.org>
AuthorDate: Tue Jul 14 16:33:48 2020 +0530
HDDS-3789. Fix TestOzoneRpcClientAbstract#testDeletedKeyForGDPR. (#1198)
(cherry picked from commit c6b45ba1574623be396b45296e2c7b7210e3c342)
---
.../org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java | 1 -
1 file changed, 1 deletion(-)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
index 591f5c6..fb17882 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
@@ -2673,7 +2673,6 @@ public abstract class TestOzoneRpcClientAbstract {
* GDPR encryption details (flag, secret, algorithm).
* @throws Exception
*/
- @Ignore
@Test
public void testDeletedKeyForGDPR() throws Exception {
//Step 1
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 14/39: HDDS-3885. Create Datanode home page (#1139)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 53f4084cc9097b477a589b6923eb0d27180b15a1
Author: maobaolong <30...@qq.com>
AuthorDate: Wed Jul 15 11:12:43 2020 +0800
HDDS-3885. Create Datanode home page (#1139)
(cherry picked from commit cbd75cbc7a0e782089c6958d159562089f42a860)
---
hadoop-hdds/container-service/pom.xml | 31 +++++++++
.../org/apache/hadoop/ozone/DNMXBean.java} | 14 +++-
.../org/apache/hadoop/ozone/DNMXBeanImpl.java} | 17 ++++-
.../apache/hadoop/ozone/HddsDatanodeService.java | 30 ++++++++-
.../webapps/hddsDatanode/dn-overview.html | 21 ++++++
.../webapps/hddsDatanode/{.gitkeep => dn.js} | 26 ++++++--
.../main/resources/webapps/hddsDatanode/index.html | 76 ++++++++++++++++++++++
.../main/resources/webapps/hddsDatanode/main.html | 20 ++++++
8 files changed, 228 insertions(+), 7 deletions(-)
diff --git a/hadoop-hdds/container-service/pom.xml b/hadoop-hdds/container-service/pom.xml
index d10d2a3..392cc44 100644
--- a/hadoop-hdds/container-service/pom.xml
+++ b/hadoop-hdds/container-service/pom.xml
@@ -107,6 +107,37 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<excludeFilterFile>${basedir}/dev-support/findbugsExcludeFile.xml</excludeFilterFile>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-common-html</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>unpack</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdds-server-framework</artifactId>
+ <outputDirectory>${project.build.outputDirectory}
+ </outputDirectory>
+ <includes>webapps/static/**/*.*</includes>
+ </artifactItem>
+ <artifactItem>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdds-docs</artifactId>
+ <outputDirectory>${project.build.outputDirectory}/webapps/hddsDatanode</outputDirectory>
+ <includes>docs/**/*.*</includes>
+ </artifactItem>
+ </artifactItems>
+ <overWriteSnapshots>true</overWriteSnapshots>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
</project>
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBean.java
similarity index 72%
copy from hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
copy to hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBean.java
index ff1232e..d36fcdb 100644
--- a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBean.java
@@ -14,4 +14,16 @@
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
- */
\ No newline at end of file
+ */
+
+package org.apache.hadoop.ozone;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.hdds.server.ServiceRuntimeInfo;
+
+/**
+ * This is the JMX management interface for DN information.
+ */
+@InterfaceAudience.Private
+public interface DNMXBean extends ServiceRuntimeInfo {
+}
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBeanImpl.java
similarity index 68%
copy from hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
copy to hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBeanImpl.java
index ff1232e..18ad66c 100644
--- a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/DNMXBeanImpl.java
@@ -14,4 +14,19 @@
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
- */
\ No newline at end of file
+ */
+
+package org.apache.hadoop.ozone;
+
+import org.apache.hadoop.hdds.server.ServiceRuntimeInfoImpl;
+import org.apache.hadoop.hdds.utils.VersionInfo;
+
+/**
+ * This is the JMX management class for DN information.
+ */
+public class DNMXBeanImpl extends ServiceRuntimeInfoImpl implements DNMXBean {
+ public DNMXBeanImpl(
+ VersionInfo versionInfo) {
+ super(versionInfo);
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
index 08eef6f..a4ff67e 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
@@ -23,12 +23,14 @@ import java.net.InetAddress;
import java.security.KeyPair;
import java.security.cert.CertificateException;
import java.util.Arrays;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.ConcurrentHashMap;
+import com.sun.jmx.mbeanserver.Introspector;
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.hdds.DFSConfigKeysLegacy;
import org.apache.hadoop.hdds.HddsUtils;
@@ -49,6 +51,7 @@ import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRe
import org.apache.hadoop.hdds.server.http.RatisDropwizardExports;
import org.apache.hadoop.hdds.tracing.TracingUtil;
import org.apache.hadoop.hdds.utils.HddsVersionInfo;
+import org.apache.hadoop.metrics2.util.MBeans;
import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils;
import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine;
import org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil;
@@ -72,6 +75,8 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import picocli.CommandLine.Command;
+import javax.management.ObjectName;
+
/**
* Datanode service plugin to start the HDDS container services.
*/
@@ -97,6 +102,10 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
private volatile AtomicBoolean isStopped = new AtomicBoolean(false);
private final Map<String, RatisDropwizardExports> ratisMetricsMap =
new ConcurrentHashMap<>();
+ private DNMXBeanImpl serviceRuntimeInfo =
+ new DNMXBeanImpl(HddsVersionInfo.HDDS_VERSION_INFO) {};
+ private ObjectName dnInfoBeanName;
+
//Constructor for DataNode PluginService
public HddsDatanodeService(){}
@@ -134,6 +143,7 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
public static void main(String[] args) {
try {
+ Introspector.checkCompliance(DNMXBeanImpl.class);
HddsDatanodeService hddsDatanodeService =
createHddsDatanodeService(args, true);
hddsDatanodeService.run(args);
@@ -182,6 +192,8 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
}
public void start() {
+ serviceRuntimeInfo.setStartTime();
+
RatisDropwizardExports.
registerRatisMetricReporters(ratisMetricsMap);
@@ -250,7 +262,7 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
.equalsIgnoreCase(System.getenv("OZONE_DATANODE_STANDALONE_TEST"))) {
startRatisForTest();
}
-
+ registerMXBean();
} catch (IOException e) {
throw new RuntimeException("Can't start the HDDS datanode plugin", e);
} catch (AuthenticationException ex) {
@@ -350,6 +362,21 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
}
}
+ private void registerMXBean() {
+ Map<String, String> jmxProperties = new HashMap<>();
+ jmxProperties.put("component", "ServerRuntime");
+ this.dnInfoBeanName = HddsUtils.registerWithJmxProperties(
+ "HddsDatanodeService",
+ "HddsDatanodeServiceInfo", jmxProperties, this.serviceRuntimeInfo);
+ }
+
+ private void unregisterMXBean() {
+ if (this.dnInfoBeanName != null) {
+ MBeans.unregister(this.dnInfoBeanName);
+ this.dnInfoBeanName = null;
+ }
+ }
+
/**
* Creates CSR for DN.
* @param config
@@ -517,6 +544,7 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
LOG.error("Stopping HttpServer is failed.", e);
}
}
+ unregisterMXBean();
}
}
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html
new file mode 100644
index 0000000..d4f7a17
--- /dev/null
+++ b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html
@@ -0,0 +1,21 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<table class="table table-bordered table-striped" class="col-md-6">
+ <tbody>
+ </tbody>
+</table>
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js
similarity index 55%
rename from hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
rename to hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js
index ff1232e..c43eb42 100644
--- a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/.gitkeep
+++ b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js
@@ -6,12 +6,30 @@
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
- */
\ No newline at end of file
+ */
+(function () {
+ "use strict";
+ angular.module('dn', ['ozone', 'nvd3']);
+
+ angular.module('dn').component('dnOverview', {
+ templateUrl: 'dn-overview.html',
+ require: {
+ overview: "^overview"
+ },
+ controller: function ($http) {
+ var ctrl = this;
+ $http.get("jmx?qry=Hadoop:service=HddsDatanode,name=StorageContainerMetrics")
+ .then(function (result) {
+ ctrl.dnmetrics = result.data.beans[0];
+ });
+ }
+ });
+})();
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/index.html b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/index.html
new file mode 100644
index 0000000..b1f703c
--- /dev/null
+++ b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/index.html
@@ -0,0 +1,76 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <meta name="description" content="HDFS Storage Container Manager">
+
+ <title>HDDS Datanode Service</title>
+
+ <link href="static/bootstrap-3.4.1/css/bootstrap.min.css" rel="stylesheet">
+ <link href="static/hadoop.css" rel="stylesheet">
+ <link href="static/nvd3-1.8.5.min.css" rel="stylesheet">
+
+ <link href="static/ozone.css" rel="stylesheet">
+
+</head>
+
+<body ng-app="dn">
+
+<header class="navbar navbar-inverse navbar-fixed-top bs-docs-nav">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar"
+ aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a class="navbar-brand" href="#">HDDS Datanode Service</a>
+ </div>
+
+
+ <navmenu
+ metrics="{ 'Rpc metrics' : '#!/metrics/rpc'}"></navmenu>
+
+
+ </div>
+</header>
+
+<div class="container-fluid" style="margin: 12pt">
+
+ <ng-view></ng-view>
+
+</div><!-- /.container -->
+
+<script src="static/jquery-3.5.1.min.js"></script>
+<script src="static/angular-1.7.9.min.js"></script>
+<script src="static/angular-route-1.7.9.min.js"></script>
+<script src="static/d3-3.5.17.min.js"></script>
+<script src="static/nvd3-1.8.5.min.js"></script>
+<script src="static/angular-nvd3-1.0.9.min.js"></script>
+<script src="static/ozone.js"></script>
+<script src="dn.js"></script>
+<script src="static/bootstrap-3.4.1/js/bootstrap.min.js"></script>
+</body>
+</html>
diff --git a/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/main.html b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/main.html
new file mode 100644
index 0000000..c639b0b
--- /dev/null
+++ b/hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/main.html
@@ -0,0 +1,20 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+<overview>
+ <dn-overview>
+ </dn-overview>
+</overview>
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 38/39: HDDS-3933. Fix memory leak because of too
many Datanode State Machine Thread (#1185)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 507bba57626fd37c51d50a8cb08784622d37e432
Author: runzhiwang <51...@users.noreply.github.com>
AuthorDate: Wed Jul 22 20:40:04 2020 +0800
HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread (#1185)
(cherry picked from commit ff7b5a3367eccc0969bfd92a2cafe48899a2aaa5)
---
.../common/statemachine/DatanodeStateMachine.java | 25 +++++-
.../common/statemachine/StateContext.java | 34 +++++++-
.../states/datanode/RunningDatanodeState.java | 14 +++-
.../common/statemachine/TestStateContext.java | 30 ++++++++
.../states/datanode/TestRunningDatanodeState.java | 90 ++++++++++++++++++++++
5 files changed, 184 insertions(+), 9 deletions(-)
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
index 779b60a..27e814b 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java
@@ -19,11 +19,13 @@ package org.apache.hadoop.ozone.container.common.statemachine;
import java.io.Closeable;
import java.io.IOException;
import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import org.apache.hadoop.hdds.HddsUtils;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.CommandStatusReportsProto;
@@ -50,7 +52,6 @@ import org.apache.hadoop.ozone.container.replication.SimpleContainerDownloader;
import org.apache.hadoop.ozone.protocol.commands.SCMCommand;
import org.apache.hadoop.util.JvmPauseMonitor;
import org.apache.hadoop.util.Time;
-import org.apache.hadoop.util.concurrent.HadoopExecutors;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
@@ -103,9 +104,10 @@ public class DatanodeStateMachine implements Closeable {
this.hddsDatanodeStopService = hddsDatanodeStopService;
this.conf = conf;
this.datanodeDetails = datanodeDetails;
- executorService = HadoopExecutors.newCachedThreadPool(
- new ThreadFactoryBuilder().setDaemon(true)
- .setNameFormat("Datanode State Machine Thread - %d").build());
+ executorService = Executors.newFixedThreadPool(
+ getEndPointTaskThreadPoolSize(),
+ new ThreadFactoryBuilder()
+ .setNameFormat("Datanode State Machine Task Thread - %d").build());
connectionManager = new SCMConnectionManager(conf);
context = new StateContext(this.conf, DatanodeStates.getInitState(), this);
// OzoneContainer instance is used in a non-thread safe way by the context
@@ -155,6 +157,21 @@ public class DatanodeStateMachine implements Closeable {
.build();
}
+ private int getEndPointTaskThreadPoolSize() {
+ // TODO(runzhiwang): current only support one recon, if support multiple
+ // recon in future reconServerCount should be the real number of recon
+ int reconServerCount = 1;
+ int totalServerCount = reconServerCount;
+
+ try {
+ totalServerCount += HddsUtils.getSCMAddresses(conf).size();
+ } catch (Exception e) {
+ LOG.error("Fail to get scm addresses", e);
+ }
+
+ return totalServerCount;
+ }
+
/**
*
* Return DatanodeDetails if set, return null otherwise.
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
index f3a599d..51262c3 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
@@ -28,6 +28,7 @@ import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicLong;
@@ -35,6 +36,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Consumer;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.CommandStatus.Status;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerAction;
@@ -51,6 +53,8 @@ import com.google.common.base.Preconditions;
import com.google.protobuf.GeneratedMessage;
import static java.lang.Math.min;
import org.apache.commons.collections.CollectionUtils;
+
+import static org.apache.hadoop.hdds.utils.HddsServerUtil.getLogWarnInterval;
import static org.apache.hadoop.hdds.utils.HddsServerUtil.getScmHeartbeatInterval;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -74,6 +78,7 @@ public class StateContext {
private DatanodeStateMachine.DatanodeStates state;
private boolean shutdownOnError = false;
private boolean shutdownGracefully = false;
+ private final AtomicLong threadPoolNotAvailableCount;
/**
* Starting with a 2 sec heartbeat frequency which will be updated to the
@@ -103,6 +108,7 @@ public class StateContext {
pipelineActions = new HashMap<>();
lock = new ReentrantLock();
stateExecutionCount = new AtomicLong(0);
+ threadPoolNotAvailableCount = new AtomicLong(0);
}
/**
@@ -393,6 +399,20 @@ public class StateContext {
}
}
+ @VisibleForTesting
+ public boolean isThreadPoolAvailable(ExecutorService executor) {
+ if (!(executor instanceof ThreadPoolExecutor)) {
+ return true;
+ }
+
+ ThreadPoolExecutor ex = (ThreadPoolExecutor) executor;
+ if (ex.getQueue().size() == 0) {
+ return true;
+ }
+
+ return false;
+ }
+
/**
* Executes the required state function.
*
@@ -415,7 +435,19 @@ public class StateContext {
if (this.isEntering()) {
task.onEnter();
}
- task.execute(service);
+
+ if (isThreadPoolAvailable(service)) {
+ task.execute(service);
+ threadPoolNotAvailableCount.set(0);
+ } else {
+ if (threadPoolNotAvailableCount.get()
+ % getLogWarnInterval(conf) == 0) {
+ LOG.warn("No available thread in pool for past {} seconds.",
+ unit.toSeconds(time) * (threadPoolNotAvailableCount.get() + 1));
+ }
+ threadPoolNotAvailableCount.incrementAndGet();
+ }
+
DatanodeStateMachine.DatanodeStates newState = task.await(time, unit);
if (this.state != newState) {
if (LOG.isDebugEnabled()) {
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/RunningDatanodeState.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/RunningDatanodeState.java
index 8a9bcaf..b0cfb4c 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/RunningDatanodeState.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/datanode/RunningDatanodeState.java
@@ -16,6 +16,7 @@
*/
package org.apache.hadoop.ozone.container.common.states.datanode;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine;
import org.apache.hadoop.ozone.container.common.statemachine.EndpointStateMachine;
@@ -42,7 +43,6 @@ import java.util.concurrent.ExecutorCompletionService;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
/**
* Class that implements handshake with SCM.
@@ -152,6 +152,11 @@ public class RunningDatanodeState implements DatanodeState {
}
}
+ @VisibleForTesting
+ public void setExecutorCompletionService(ExecutorCompletionService e) {
+ this.ecs = e;
+ }
+
private Callable<EndPointStates> getEndPointTask(
EndpointStateMachine endpoint) {
if (endpointTasks.containsKey(endpoint)) {
@@ -200,10 +205,11 @@ public class RunningDatanodeState implements DatanodeState {
@Override
public DatanodeStateMachine.DatanodeStates
await(long duration, TimeUnit timeUnit)
- throws InterruptedException, ExecutionException, TimeoutException {
+ throws InterruptedException {
int count = connectionManager.getValues().size();
int returned = 0;
- long timeLeft = timeUnit.toMillis(duration);
+ long durationMS = timeUnit.toMillis(duration);
+ long timeLeft = durationMS;
long startTime = Time.monotonicNow();
List<Future<EndPointStates>> results = new LinkedList<>();
@@ -214,7 +220,7 @@ public class RunningDatanodeState implements DatanodeState {
results.add(result);
returned++;
}
- timeLeft = timeLeft - (Time.monotonicNow() - startTime);
+ timeLeft = durationMS - (Time.monotonicNow() - startTime);
}
return computeNextContainerState(results);
}
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/TestStateContext.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/TestStateContext.java
index 545d670..c3fd310 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/TestStateContext.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/TestStateContext.java
@@ -28,6 +28,8 @@ import static org.mockito.Mockito.mock;
import java.net.InetSocketAddress;
import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
@@ -39,6 +41,8 @@ import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolPro
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineAction;
import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.DatanodeStates;
import org.apache.hadoop.ozone.container.common.states.DatanodeState;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.junit.Assert;
import org.junit.Test;
import com.google.protobuf.GeneratedMessage;
@@ -182,4 +186,30 @@ public class TestStateContext {
assertEquals(DatanodeStates.SHUTDOWN, subject.getState());
}
+ @Test
+ public void testIsThreadPoolAvailable() throws Exception {
+ StateContext stateContext = new StateContext(null, null, null);
+
+ int threadPoolSize = 2;
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ threadPoolSize);
+
+ CompletableFuture<String> futureOne = new CompletableFuture<>();
+ CompletableFuture<String> futureTwo = new CompletableFuture<>();
+
+ // task num greater than pool size
+ for (int i = 0; i < threadPoolSize; i++) {
+ executorService.submit(() -> futureOne.get());
+ }
+ executorService.submit(() -> futureTwo.get());
+
+ Assert.assertFalse(stateContext.isThreadPoolAvailable(executorService));
+
+ futureOne.complete("futureOne");
+ LambdaTestUtils.await(1000, 100, () ->
+ stateContext.isThreadPoolAvailable(executorService));
+
+ futureTwo.complete("futureTwo");
+ executorService.shutdown();
+ }
}
\ No newline at end of file
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/datanode/TestRunningDatanodeState.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/datanode/TestRunningDatanodeState.java
new file mode 100644
index 0000000..9fb4307
--- /dev/null
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/datanode/TestRunningDatanodeState.java
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.container.common.states.datanode;
+
+import org.apache.hadoop.ozone.container.common.statemachine.EndpointStateMachine;
+import org.apache.hadoop.ozone.container.common.statemachine.SCMConnectionManager;
+import org.apache.hadoop.util.Time;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hadoop.ozone.container.common.statemachine.EndpointStateMachine.EndPointStates.SHUTDOWN;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test class for RunningDatanodeState.
+ */
+public class TestRunningDatanodeState {
+ @Test
+ public void testAwait() throws InterruptedException {
+ SCMConnectionManager connectionManager =
+ Mockito.mock(SCMConnectionManager.class);
+ List<EndpointStateMachine> stateMachines = new ArrayList<>();
+ when(connectionManager.getValues()).thenReturn(stateMachines);
+
+ RunningDatanodeState state =
+ new RunningDatanodeState(null, connectionManager, null);
+
+ int threadPoolSize = 2;
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ threadPoolSize);
+
+ ExecutorCompletionService ecs =
+ new ExecutorCompletionService<>(executorService);
+ state.setExecutorCompletionService(ecs);
+
+ for (int i = 0; i < threadPoolSize; i++) {
+ stateMachines.add(new EndpointStateMachine(null, null, null));
+ }
+
+ CompletableFuture<EndpointStateMachine.EndPointStates> futureOne =
+ new CompletableFuture<>();
+ for (int i = 0; i < threadPoolSize; i++) {
+ ecs.submit(() -> futureOne.get());
+ }
+
+ long startTime = Time.monotonicNow();
+ state.await(500, TimeUnit.MILLISECONDS);
+ long endTime = Time.monotonicNow();
+ Assert.assertTrue((endTime - startTime) >= 500);
+
+ futureOne.complete(SHUTDOWN);
+
+ CompletableFuture<EndpointStateMachine.EndPointStates> futureTwo =
+ new CompletableFuture<>();
+ for (int i = 0; i < threadPoolSize; i++) {
+ ecs.submit(() -> futureTwo.get());
+ }
+ futureTwo.complete(SHUTDOWN);
+
+ startTime = Time.monotonicNow();
+ state.await(500, TimeUnit.MILLISECONDS);
+ endTime = Time.monotonicNow();
+ Assert.assertTrue((endTime - startTime) < 500);
+
+ executorService.shutdown();
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 29/39: HDDS-3741. Reload old OM state if Install
Snapshot from Leader fails (#1129)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit d92bcba1c6a2b801c04a69807fc91f01895136bd
Author: Hanisha Koneru <ha...@apache.org>
AuthorDate: Fri Jul 17 13:54:08 2020 -0700
HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (#1129)
(cherry picked from commit 3571d7e72688b008c2667997cb4d824ccc9a8a3e)
---
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 3 +
.../org/apache/hadoop/ozone/util/ExitManager.java | 33 ++++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 215 ++++++++++++++++-----
.../snapshot/TestOzoneManagerSnapshotProvider.java | 26 +--
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 23 ++-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 176 ++++++++++-------
.../hadoop/ozone/om/ratis/OMTransactionInfo.java | 21 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 6 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 37 +++-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 10 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 2 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 2 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 2 +-
13 files changed, 399 insertions(+), 157 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
index e340b32..4b38094 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
@@ -366,4 +366,7 @@ public final class OzoneConsts {
public static final String CONTAINER_DB_TYPE_ROCKSDB = "RocksDB";
public static final String CONTAINER_DB_TYPE_LEVELDB = "LevelDB";
+
+ // An on-disk transient marker file used when replacing DB with checkpoint
+ public static final String DB_TRANSIENT_MARKER = "dbInconsistentMarker";
}
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/ExitManager.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/ExitManager.java
new file mode 100644
index 0000000..4a83c1d
--- /dev/null
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/ExitManager.java
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.util;
+
+import org.apache.ratis.util.ExitUtils;
+import org.slf4j.Logger;
+
+/**
+ * An Exit Manager used to shutdown service in case of unrecoverable error.
+ * This class will be helpful to test exit functionality.
+ */
+public class ExitManager {
+
+ public void exitSystem(int status, String message, Throwable throwable,
+ Logger log) {
+ ExitUtils.terminate(1, message, throwable, log);
+ }
+}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java
index ea422e1..1465d95 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java
@@ -18,10 +18,12 @@ package org.apache.hadoop.ozone.om;
import java.io.File;
import java.io.IOException;
+import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
+import org.apache.commons.lang3.RandomStringUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
import org.apache.hadoop.ozone.MiniOzoneCluster;
@@ -33,10 +35,14 @@ import org.apache.hadoop.ozone.client.OzoneVolume;
import org.apache.hadoop.ozone.client.VolumeArgs;
import org.apache.hadoop.ozone.om.ratis.OMTransactionInfo;
import org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer;
+import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils;
+import org.apache.hadoop.ozone.util.ExitManager;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.protocol.TermIndex;
-import org.apache.commons.lang3.RandomStringUtils;
import static org.apache.hadoop.ozone.om.TestOzoneManagerHAWithData.createKey;
-import org.apache.ratis.server.protocol.TermIndex;
+import static org.junit.Assert.assertTrue;
+
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -44,6 +50,8 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.event.Level;
/**
* Tests the Ratis snaphsots feature in OM.
@@ -57,6 +65,10 @@ public class TestOMRatisSnapshots {
private String scmId;
private String omServiceId;
private int numOfOMs = 3;
+ private OzoneBucket ozoneBucket;
+ private String volumeName;
+ private String bucketName;
+
private static final long SNAPSHOT_THRESHOLD = 50;
private static final int LOG_PURGE_GAP = 50;
@@ -93,6 +105,20 @@ public class TestOMRatisSnapshots {
cluster.waitForClusterToBeReady();
objectStore = OzoneClientFactory.getRpcClient(omServiceId, conf)
.getObjectStore();
+
+ volumeName = "volume" + RandomStringUtils.randomNumeric(5);
+ bucketName = "bucket" + RandomStringUtils.randomNumeric(5);
+
+ VolumeArgs createVolumeArgs = VolumeArgs.newBuilder()
+ .setOwner("user" + RandomStringUtils.randomNumeric(5))
+ .setAdmin("admin" + RandomStringUtils.randomNumeric(5))
+ .build();
+
+ objectStore.createVolume(volumeName, createVolumeArgs);
+ OzoneVolume retVolumeinfo = objectStore.getVolume(volumeName);
+
+ retVolumeinfo.createBucket(bucketName);
+ ozoneBucket = retVolumeinfo.getBucket(bucketName);
}
/**
@@ -123,37 +149,13 @@ public class TestOMRatisSnapshots {
OzoneManager followerOM = cluster.getOzoneManager(followerNodeId);
// Do some transactions so that the log index increases
- String userName = "user" + RandomStringUtils.randomNumeric(5);
- String adminName = "admin" + RandomStringUtils.randomNumeric(5);
- String volumeName = "volume" + RandomStringUtils.randomNumeric(5);
- String bucketName = "bucket" + RandomStringUtils.randomNumeric(5);
-
- VolumeArgs createVolumeArgs = VolumeArgs.newBuilder()
- .setOwner(userName)
- .setAdmin(adminName)
- .build();
-
- objectStore.createVolume(volumeName, createVolumeArgs);
- OzoneVolume retVolumeinfo = objectStore.getVolume(volumeName);
-
- retVolumeinfo.createBucket(bucketName);
- OzoneBucket ozoneBucket = retVolumeinfo.getBucket(bucketName);
-
- long leaderOMappliedLogIndex =
- leaderRatisServer.getLastAppliedTermIndex().getIndex();
-
- List<String> keys = new ArrayList<>();
- while (leaderOMappliedLogIndex < 2000) {
- keys.add(createKey(ozoneBucket));
- leaderOMappliedLogIndex =
- leaderRatisServer.getLastAppliedTermIndex().getIndex();
- }
+ List<String> keys = writeKeysToIncreaseLogIndex(leaderRatisServer, 200);
// Get the latest db checkpoint from the leader OM.
OMTransactionInfo omTransactionInfo =
OMTransactionInfo.readTransactionInfo(leaderOM.getMetadataManager());
TermIndex leaderOMTermIndex =
- TermIndex.newTermIndex(omTransactionInfo.getCurrentTerm(),
+ TermIndex.newTermIndex(omTransactionInfo.getTerm(),
omTransactionInfo.getTransactionIndex());
long leaderOMSnaphsotIndex = leaderOMTermIndex.getIndex();
long leaderOMSnapshotTermIndex = leaderOMTermIndex.getTerm();
@@ -167,30 +169,20 @@ public class TestOMRatisSnapshots {
// The recently started OM should be lagging behind the leader OM.
long followerOMLastAppliedIndex =
followerOM.getOmRatisServer().getLastAppliedTermIndex().getIndex();
- Assert.assertTrue(
+ assertTrue(
followerOMLastAppliedIndex < leaderOMSnaphsotIndex);
// Install leader OM's db checkpoint on the lagging OM.
- File oldDbLocation = followerOM.getMetadataManager().getStore()
- .getDbLocation();
- followerOM.getOmRatisServer().getOmStateMachine().pause();
- followerOM.getMetadataManager().getStore().close();
- followerOM.replaceOMDBWithCheckpoint(leaderOMSnaphsotIndex, oldDbLocation,
- leaderDbCheckpoint.getCheckpointLocation());
-
- // Reload the follower OM with new DB checkpoint from the leader OM.
- followerOM.reloadOMState(leaderOMSnaphsotIndex, leaderOMSnapshotTermIndex);
- followerOM.getOmRatisServer().getOmStateMachine().unpause(
- leaderOMSnaphsotIndex, leaderOMSnapshotTermIndex);
-
- // After the new checkpoint is loaded and state machine is unpaused, the
- // follower OM lastAppliedIndex must match the snapshot index of the
- // checkpoint.
+ followerOM.installCheckpoint(leaderOMNodeId, leaderDbCheckpoint);
+
+ // After the new checkpoint is installed, the follower OM
+ // lastAppliedIndex must >= the snapshot index of the checkpoint. It
+ // could be great than snapshot index if there is any conf entry from ratis.
followerOMLastAppliedIndex = followerOM.getOmRatisServer()
.getLastAppliedTermIndex().getIndex();
- Assert.assertEquals(leaderOMSnaphsotIndex, followerOMLastAppliedIndex);
- Assert.assertEquals(leaderOMSnapshotTermIndex,
- followerOM.getOmRatisServer().getLastAppliedTermIndex().getTerm());
+ assertTrue(followerOMLastAppliedIndex >= leaderOMSnaphsotIndex);
+ assertTrue(followerOM.getOmRatisServer().getLastAppliedTermIndex()
+ .getTerm() >= leaderOMSnapshotTermIndex);
// Verify that the follower OM's DB contains the transactions which were
// made while it was inactive.
@@ -204,4 +196,133 @@ public class TestOMRatisSnapshots {
followerOMMetaMngr.getOzoneKey(volumeName, bucketName, key)));
}
}
+
+ @Test
+ public void testInstallOldCheckpointFailure() throws Exception {
+ // Get the leader OM
+ String leaderOMNodeId = OmFailoverProxyUtil
+ .getFailoverProxyProvider(objectStore.getClientProxy())
+ .getCurrentProxyOMNodeId();
+
+ OzoneManager leaderOM = cluster.getOzoneManager(leaderOMNodeId);
+
+ // Find the inactive OM and start it
+ String followerNodeId = leaderOM.getPeerNodes().get(0).getOMNodeId();
+ if (cluster.isOMActive(followerNodeId)) {
+ followerNodeId = leaderOM.getPeerNodes().get(1).getOMNodeId();
+ }
+ cluster.startInactiveOM(followerNodeId);
+
+ OzoneManager followerOM = cluster.getOzoneManager(followerNodeId);
+ OzoneManagerRatisServer followerRatisServer = followerOM.getOmRatisServer();
+
+ // Do some transactions so that the log index increases on follower OM
+ writeKeysToIncreaseLogIndex(followerRatisServer, 100);
+
+ TermIndex leaderCheckpointTermIndex = leaderOM.getOmRatisServer()
+ .getLastAppliedTermIndex();
+ DBCheckpoint leaderDbCheckpoint = leaderOM.getMetadataManager().getStore()
+ .getCheckpoint(false);
+
+ // Do some more transactions to increase the log index further on
+ // follower OM such that it is more than the checkpoint index taken on
+ // leader OM.
+ writeKeysToIncreaseLogIndex(followerOM.getOmRatisServer(),
+ leaderCheckpointTermIndex.getIndex() + 100);
+
+ GenericTestUtils.setLogLevel(OzoneManager.LOG, Level.INFO);
+ GenericTestUtils.LogCapturer logCapture =
+ GenericTestUtils.LogCapturer.captureLogs(OzoneManager.LOG);
+
+ // Install the old checkpoint on the follower OM. This should fail as the
+ // followerOM is already ahead of that transactionLogIndex and the OM
+ // state should be reloaded.
+ TermIndex followerTermIndex = followerRatisServer.getLastAppliedTermIndex();
+ TermIndex newTermIndex = followerOM.installCheckpoint(
+ leaderOMNodeId, leaderDbCheckpoint);
+
+ String errorMsg = "Cannot proceed with InstallSnapshot as OM is at " +
+ "TermIndex " + followerTermIndex + " and checkpoint has lower " +
+ "TermIndex";
+ Assert.assertTrue(logCapture.getOutput().contains(errorMsg));
+ Assert.assertNull("OM installed checkpoint even though checkpoint " +
+ "logIndex is less than it's lastAppliedIndex", newTermIndex);
+ Assert.assertEquals(followerTermIndex,
+ followerRatisServer.getLastAppliedTermIndex());
+ }
+
+ @Test
+ public void testInstallCorruptedCheckpointFailure() throws Exception {
+ // Get the leader OM
+ String leaderOMNodeId = OmFailoverProxyUtil
+ .getFailoverProxyProvider(objectStore.getClientProxy())
+ .getCurrentProxyOMNodeId();
+
+ OzoneManager leaderOM = cluster.getOzoneManager(leaderOMNodeId);
+ OzoneManagerRatisServer leaderRatisServer = leaderOM.getOmRatisServer();
+
+ // Find the inactive OM
+ String followerNodeId = leaderOM.getPeerNodes().get(0).getOMNodeId();
+ if (cluster.isOMActive(followerNodeId)) {
+ followerNodeId = leaderOM.getPeerNodes().get(1).getOMNodeId();
+ }
+ OzoneManager followerOM = cluster.getOzoneManager(followerNodeId);
+ OzoneManagerRatisServer followerRatisServer = followerOM.getOmRatisServer();
+
+ // Do some transactions so that the log index increases
+ writeKeysToIncreaseLogIndex(leaderRatisServer, 100);
+
+ DBCheckpoint leaderDbCheckpoint = leaderOM.getMetadataManager().getStore()
+ .getCheckpoint(false);
+ Path leaderCheckpointLocation = leaderDbCheckpoint.getCheckpointLocation();
+ OMTransactionInfo leaderCheckpointTrxnInfo = OzoneManagerRatisUtils
+ .getTrxnInfoFromCheckpoint(conf, leaderCheckpointLocation);
+
+ // Corrupt the leader checkpoint and install that on the OM. The
+ // operation should fail and OM should shutdown.
+ boolean delete = true;
+ for (File file : leaderCheckpointLocation.toFile()
+ .listFiles()) {
+ if (file.getName().contains(".sst")) {
+ if (delete) {
+ file.delete();
+ delete = false;
+ } else {
+ delete = true;
+ }
+ }
+ }
+
+ GenericTestUtils.setLogLevel(OzoneManager.LOG, Level.ERROR);
+ GenericTestUtils.LogCapturer logCapture =
+ GenericTestUtils.LogCapturer.captureLogs(OzoneManager.LOG);
+ followerOM.setExitManagerForTesting(new DummyExitManager());
+
+ followerOM.installCheckpoint(leaderOMNodeId, leaderCheckpointLocation,
+ leaderCheckpointTrxnInfo);
+
+ Assert.assertTrue(logCapture.getOutput().contains("System Exit: " +
+ "Failed to reload OM state and instantiate services."));
+ }
+
+ private List<String> writeKeysToIncreaseLogIndex(
+ OzoneManagerRatisServer omRatisServer, long targetLogIndex)
+ throws IOException, InterruptedException {
+ List<String> keys = new ArrayList<>();
+ long logIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
+ while (logIndex < targetLogIndex) {
+ keys.add(createKey(ozoneBucket));
+ Thread.sleep(100);
+ logIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
+ }
+ return keys;
+ }
+
+ private class DummyExitManager extends ExitManager {
+ @Override
+ public void exitSystem(int status, String message, Throwable throwable,
+ Logger log) {
+ log.error("System Exit: " + message, throwable);
+ }
+ }
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerSnapshotProvider.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerSnapshotProvider.java
index d77f4d9..844c859 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerSnapshotProvider.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerSnapshotProvider.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.ozone.om.snapshot;
import java.util.UUID;
+import org.apache.commons.lang3.RandomStringUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
import org.apache.hadoop.ozone.MiniOzoneCluster;
@@ -31,11 +32,10 @@ import org.apache.hadoop.ozone.client.OzoneVolume;
import org.apache.hadoop.ozone.client.VolumeArgs;
import org.apache.hadoop.ozone.om.OMConfigKeys;
import org.apache.hadoop.ozone.om.OmFailoverProxyUtil;
-import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
import org.apache.hadoop.ozone.om.OzoneManager;
-
-import org.apache.commons.lang3.RandomStringUtils;
import org.apache.hadoop.ozone.om.ratis.OMTransactionInfo;
+import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils;
+
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -124,7 +124,7 @@ public class TestOzoneManagerSnapshotProvider {
.getOzoneManagerDBSnapshot(leaderOMNodeId);
long leaderSnapshotIndex = leaderOM.getRatisSnapshotIndex();
- long downloadedSnapshotIndex = getDownloadSnapshotIndex(omSnapshot);
+ long downloadedSnapshotIndex = getDownloadedSnapshotIndex(omSnapshot);
// The snapshot index downloaded from leader OM should match the ratis
// snapshot index on the leader OM
@@ -133,21 +133,13 @@ public class TestOzoneManagerSnapshotProvider {
leaderSnapshotIndex, downloadedSnapshotIndex);
}
- private long getDownloadSnapshotIndex(DBCheckpoint dbCheckpoint)
+ private long getDownloadedSnapshotIndex(DBCheckpoint dbCheckpoint)
throws Exception {
- OzoneConfiguration configuration = new OzoneConfiguration(conf);
- configuration.set(OMConfigKeys.OZONE_OM_DB_DIRS,
- dbCheckpoint.getCheckpointLocation().getParent().toString());
-
- OmMetadataManagerImpl omMetadataManager =
- new OmMetadataManagerImpl(configuration);
-
- long transactionIndex =
- OMTransactionInfo.readTransactionInfo(omMetadataManager)
- .getTransactionIndex();
- omMetadataManager.stop();
- return transactionIndex;
+ OMTransactionInfo trxnInfoFromCheckpoint =
+ OzoneManagerRatisUtils.getTrxnInfoFromCheckpoint(conf,
+ dbCheckpoint.getCheckpointLocation());
+ return trxnInfoFromCheckpoint.getTransactionIndex();
}
}
\ No newline at end of file
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
index d48c6fa..6c8b505 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
@@ -77,9 +77,11 @@ import com.google.common.collect.Lists;
import org.apache.commons.lang3.StringUtils;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS_DEFAULT;
+import static org.apache.hadoop.ozone.OzoneConsts.DB_TRANSIENT_MARKER;
import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import org.apache.ratis.util.ExitUtils;
import org.eclipse.jetty.util.StringUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -249,6 +251,20 @@ public class OmMetadataManagerImpl implements OMMetadataManager {
if (store == null) {
File metaDir = OMStorage.getOmDbDir(configuration);
+ // Check if there is a DB Inconsistent Marker in the metaDir. This
+ // marker indicates that the DB is in an inconsistent state and hence
+ // the OM process should be terminated.
+ File markerFile = new File(metaDir, DB_TRANSIENT_MARKER);
+ if (markerFile.exists()) {
+ LOG.error("File {} marks that OM DB is in an inconsistent state.");
+ // Note - The marker file should be deleted only after fixing the DB.
+ // In an HA setup, this can be done by replacing this DB with a
+ // checkpoint from another OM.
+ String errorMsg = "Cannot load OM DB as it is in an inconsistent " +
+ "state.";
+ ExitUtils.terminate(1, errorMsg, LOG);
+ }
+
RocksDBConfiguration rocksDBConfiguration =
configuration.getObject(RocksDBConfiguration.class);
@@ -273,10 +289,15 @@ public class OmMetadataManagerImpl implements OMMetadataManager {
public static DBStore loadDB(OzoneConfiguration configuration, File metaDir)
throws IOException {
+ return loadDB(configuration, metaDir, OM_DB_NAME);
+ }
+
+ public static DBStore loadDB(OzoneConfiguration configuration, File metaDir,
+ String dbName) throws IOException {
RocksDBConfiguration rocksDBConfiguration =
configuration.getObject(RocksDBConfiguration.class);
DBStoreBuilder dbStoreBuilder = DBStoreBuilder.newBuilder(configuration,
- rocksDBConfiguration).setName(OM_DB_NAME)
+ rocksDBConfiguration).setName(dbName)
.setPath(Paths.get(metaDir.getPath()));
DBStore dbStore = addOMTablesAndCodecs(dbStoreBuilder).build();
return dbStore;
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 0905d81..8a49fa7 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@ -155,6 +155,7 @@ import org.apache.hadoop.ozone.security.acl.OzoneObj.ResourceType;
import org.apache.hadoop.ozone.security.acl.OzoneObj.StoreType;
import org.apache.hadoop.ozone.security.acl.OzoneObjInfo;
import org.apache.hadoop.ozone.security.acl.RequestContext;
+import org.apache.hadoop.ozone.util.ExitManager;
import org.apache.hadoop.ozone.util.OzoneVersionInfo;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
@@ -194,6 +195,7 @@ import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_KEY_PREALLOCATION_BL
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_KEY_PREALLOCATION_BLOCKS_MAX_DEFAULT;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE_DEFAULT;
+import static org.apache.hadoop.ozone.OzoneConsts.DB_TRANSIENT_MARKER;
import static org.apache.hadoop.ozone.OzoneConsts.OM_METRICS_FILE;
import static org.apache.hadoop.ozone.OzoneConsts.OM_METRICS_TEMP_FILE;
import static org.apache.hadoop.ozone.OzoneConsts.RPC_PORT;
@@ -217,6 +219,7 @@ import static org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.
import org.apache.ratis.proto.RaftProtos.RaftPeerRole;
import org.apache.ratis.server.protocol.TermIndex;
+import org.apache.ratis.util.ExitUtils;
import org.apache.ratis.util.FileUtils;
import org.apache.ratis.util.LifeCycle;
import org.bouncycastle.pkcs.PKCS10CertificationRequest;
@@ -308,6 +311,8 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
private boolean isNativeAuthorizerEnabled;
+ private ExitManager exitManager;
+
private enum State {
INITIALIZED,
RUNNING,
@@ -3066,51 +3071,52 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
/**
* Download and install latest checkpoint from leader OM.
- * If the download checkpoints snapshot index is greater than this OM's
- * last applied transaction index, then re-initialize the OM state via this
- * checkpoint. Before re-initializing OM state, the OM Ratis server should
- * be stopped so that no new transactions can be applied.
*
* @param leaderId peerNodeID of the leader OM
- * @return If checkpoint is installed, return the corresponding termIndex.
- * Otherwise, return null.
+ * @return If checkpoint is installed successfully, return the
+ * corresponding termIndex. Otherwise, return null.
*/
- public TermIndex installSnapshot(String leaderId) {
+ public TermIndex installSnapshotFromLeader(String leaderId) {
if (omSnapshotProvider == null) {
LOG.error("OM Snapshot Provider is not configured as there are no peer " +
"nodes.");
return null;
}
- DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
- Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
+ DBCheckpoint omDBCheckpoint = getDBCheckpointFromLeader(leaderId);
+ LOG.info("Downloaded checkpoint from Leader {} to the location {}",
+ leaderId, omDBCheckpoint.getCheckpointLocation());
- LOG.info("Downloaded checkpoint from Leader {}, in to the location {}",
- leaderId, newDBlocation);
+ TermIndex termIndex = null;
+ try {
+ termIndex = installCheckpoint(leaderId, omDBCheckpoint);
+ } catch (Exception ex) {
+ LOG.error("Failed to install snapshot from Leader OM: {}", ex);
+ }
+ return termIndex;
+ }
- // Check if current ratis log index is smaller than the downloaded
- // checkpoint transaction index. If yes, proceed by stopping the ratis
- // server so that the OM state can be re-initialized. If no, then do not
- // proceed with installSnapshot.
+ /**
+ * Install checkpoint. If the checkpoints snapshot index is greater than
+ * OM's last applied transaction index, then re-initialize the OM
+ * state via this checkpoint. Before re-initializing OM state, the OM Ratis
+ * server should be stopped so that no new transactions can be applied.
+ */
+ TermIndex installCheckpoint(String leaderId, DBCheckpoint omDBCheckpoint)
+ throws Exception {
- OMTransactionInfo omTransactionInfo = null;
+ Path checkpointLocation = omDBCheckpoint.getCheckpointLocation();
+ OMTransactionInfo checkpointTrxnInfo = OzoneManagerRatisUtils
+ .getTrxnInfoFromCheckpoint(configuration, checkpointLocation);
- Path dbDir = newDBlocation.getParent();
- if (dbDir == null) {
- LOG.error("Incorrect DB location path {} received from checkpoint.",
- newDBlocation);
- return null;
- }
+ LOG.info("Installing checkpoint with OMTransactionInfo {}",
+ checkpointTrxnInfo);
- try {
- omTransactionInfo =
- OzoneManagerRatisUtils.getTransactionInfoFromDownloadedSnapshot(
- configuration, dbDir);
- } catch (Exception ex) {
- LOG.error("Failed during opening downloaded snapshot from " +
- "{} to obtain transaction index", newDBlocation, ex);
- return null;
- }
+ return installCheckpoint(leaderId, checkpointLocation, checkpointTrxnInfo);
+ }
+
+ TermIndex installCheckpoint(String leaderId, Path checkpointLocation,
+ OMTransactionInfo checkpointTrxnInfo) throws Exception {
File oldDBLocation = metadataManager.getStore().getDbLocation();
try {
@@ -3123,58 +3129,74 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
omRatisServer.getOmStateMachine().pause();
} catch (Exception e) {
LOG.error("Failed to stop/ pause the services. Cannot proceed with " +
- "installing the new checkpoint.", e);
- return null;
- }
-
- //TODO: un-pause SM if any failures and retry?
-
- long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-
- boolean canProceed =
- OzoneManagerRatisUtils.verifyTransactionInfo(omTransactionInfo,
- lastAppliedIndex, leaderId, newDBlocation);
-
- // If downloaded DB has transaction info less than current one, return.
- if (!canProceed) {
- return null;
+ "installing the new checkpoint.");
+ // During stopServices, if KeyManager was stopped successfully and
+ // OMMetadataManager stop failed, we should restart the KeyManager.
+ keyManager.start(configuration);
+ throw e;
}
- long leaderIndex = omTransactionInfo.getTransactionIndex();
- long leaderTerm = omTransactionInfo.getCurrentTerm();
+ File dbBackup = null;
+ TermIndex termIndex = omRatisServer.getLastAppliedTermIndex();
+ long term = termIndex.getTerm();
+ long lastAppliedIndex = termIndex.getIndex();
+ // Check if current applied log index is smaller than the downloaded
+ // checkpoint transaction index. If yes, proceed by stopping the ratis
+ // server so that the OM state can be re-initialized. If no then do not
+ // proceed with installSnapshot.
+ boolean canProceed = OzoneManagerRatisUtils.verifyTransactionInfo(
+ checkpointTrxnInfo, lastAppliedIndex, leaderId, checkpointLocation);
- File dbBackup;
- try {
- dbBackup = replaceOMDBWithCheckpoint(lastAppliedIndex, oldDBLocation,
- newDBlocation);
- } catch (Exception e) {
- LOG.error("OM DB checkpoint replacement with new downloaded checkpoint " +
- "failed.", e);
- return null;
+ if (canProceed) {
+ try {
+ dbBackup = replaceOMDBWithCheckpoint(lastAppliedIndex, oldDBLocation,
+ checkpointLocation);
+ term = checkpointTrxnInfo.getTerm();
+ lastAppliedIndex = checkpointTrxnInfo.getTransactionIndex();
+ LOG.info("Replaced DB with checkpoint from OM: {}, term: {}, index: {}",
+ leaderId, term, lastAppliedIndex);
+ } catch (Exception e) {
+ LOG.error("Failed to install Snapshot from {} as OM failed to replace" +
+ " DB with downloaded checkpoint. Reloading old OM state.", e);
+ }
+ } else {
+ LOG.warn("Cannot proceed with InstallSnapshot as OM is at TermIndex {} " +
+ "and checkpoint has lower TermIndex {}. Reloading old state of OM.",
+ termIndex, checkpointTrxnInfo.getTermIndex());
}
// Reload the OM DB store with the new checkpoint.
// Restart (unpause) the state machine and update its last applied index
// to the installed checkpoint's snapshot index.
try {
- reloadOMState(leaderIndex, leaderTerm);
- omRatisServer.getOmStateMachine().unpause(leaderIndex, leaderTerm);
- } catch (IOException e) {
- LOG.error("Failed to reload OM state with new DB checkpoint.", e);
- return null;
+ reloadOMState(lastAppliedIndex, term);
+ omRatisServer.getOmStateMachine().unpause(lastAppliedIndex, term);
+ LOG.info("Reloaded OM state with Term: {} and Index: {}", term,
+ lastAppliedIndex);
+ } catch (IOException ex) {
+ String errorMsg = "Failed to reload OM state and instantiate services.";
+ exitManager.exitSystem(1, errorMsg, ex, LOG);
}
// Delete the backup DB
try {
- FileUtils.deleteFully(dbBackup);
+ if (dbBackup != null) {
+ FileUtils.deleteFully(dbBackup);
+ }
} catch (IOException e) {
LOG.error("Failed to delete the backup of the original DB {}", dbBackup);
}
+ if (lastAppliedIndex != checkpointTrxnInfo.getTransactionIndex()) {
+ // Install Snapshot failed and old state was reloaded. Return null to
+ // Ratis to indicate that installation failed.
+ return null;
+ }
+
// TODO: We should only return the snpashotIndex to the leader.
// Should be fixed after RATIS-586
- TermIndex newTermIndex = TermIndex.newTermIndex(leaderTerm, leaderIndex);
+ TermIndex newTermIndex = TermIndex.newTermIndex(term, lastAppliedIndex);
return newTermIndex;
}
@@ -3208,16 +3230,17 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
*
* @param lastAppliedIndex the last applied index in the current OM DB.
* @param checkpointPath path to the new DB checkpoint
- * @return location of the backup of the original DB
+ * @return location of backup of the original DB
* @throws Exception
*/
File replaceOMDBWithCheckpoint(long lastAppliedIndex, File oldDB,
- Path checkpointPath) throws Exception {
+ Path checkpointPath) throws IOException {
// Take a backup of the current DB
String dbBackupName = OzoneConsts.OM_DB_BACKUP_PREFIX +
lastAppliedIndex + "_" + System.currentTimeMillis();
- File dbBackup = new File(oldDB.getParentFile(), dbBackupName);
+ File dbDir = oldDB.getParentFile();
+ File dbBackup = new File(dbDir, dbBackupName);
try {
Files.move(oldDB.toPath(), dbBackup.toPath());
@@ -3228,13 +3251,28 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
}
// Move the new DB checkpoint into the om metadata dir
+ Path markerFile = new File(dbDir, DB_TRANSIENT_MARKER).toPath();
try {
+ // Create a Transient Marker file. This file will be deleted if the
+ // checkpoint DB is successfully moved to the old DB location or if the
+ // old DB backup is reset to its location. If not, then the OM DB is in
+ // an inconsistent state and this marker file will fail OM from
+ // starting up.
+ Files.createFile(markerFile);
Files.move(checkpointPath, oldDB.toPath());
+ Files.deleteIfExists(markerFile);
} catch (IOException e) {
LOG.error("Failed to move downloaded DB checkpoint {} to metadata " +
"directory {}. Resetting to original DB.", checkpointPath,
oldDB.toPath());
- Files.move(dbBackup.toPath(), oldDB.toPath());
+ try {
+ Files.move(dbBackup.toPath(), oldDB.toPath());
+ Files.deleteIfExists(markerFile);
+ } catch (IOException ex) {
+ String errorMsg = "Failed to reset to original DB. OM is in an " +
+ "inconsistent state.";
+ ExitUtils.terminate(1, errorMsg, ex, LOG);
+ }
throw e;
}
return dbBackup;
@@ -3452,4 +3490,8 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
visited);
}
+ @VisibleForTesting
+ void setExitManagerForTesting(ExitManager exitManagerForTesting) {
+ this.exitManager = exitManagerForTesting;
+ }
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMTransactionInfo.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMTransactionInfo.java
index 2441751..28c8c3a 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMTransactionInfo.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMTransactionInfo.java
@@ -24,6 +24,7 @@ import org.apache.hadoop.ozone.om.OMMetadataManager;
import java.io.IOException;
import java.util.Objects;
+import org.apache.ratis.server.protocol.TermIndex;
import static org.apache.hadoop.ozone.OzoneConsts.TRANSACTION_INFO_KEY;
import static org.apache.hadoop.ozone.OzoneConsts.TRANSACTION_INFO_SPLIT_KEY;
@@ -33,7 +34,7 @@ import static org.apache.hadoop.ozone.OzoneConsts.TRANSACTION_INFO_SPLIT_KEY;
*/
public final class OMTransactionInfo {
- private long currentTerm; // term associated with the ratis log index.
+ private long term; // term associated with the ratis log index.
// Transaction index corresponds to ratis log index
private long transactionIndex;
@@ -43,12 +44,12 @@ public final class OMTransactionInfo {
Preconditions.checkState(tInfo.length==2,
"Incorrect TransactionInfo value");
- currentTerm = Long.parseLong(tInfo[0]);
+ term = Long.parseLong(tInfo[0]);
transactionIndex = Long.parseLong(tInfo[1]);
}
private OMTransactionInfo(long currentTerm, long transactionIndex) {
- this.currentTerm = currentTerm;
+ this.term = currentTerm;
this.transactionIndex = transactionIndex;
}
@@ -56,8 +57,8 @@ public final class OMTransactionInfo {
* Get current term.
* @return currentTerm
*/
- public long getCurrentTerm() {
- return currentTerm;
+ public long getTerm() {
+ return term;
}
/**
@@ -68,6 +69,10 @@ public final class OMTransactionInfo {
return transactionIndex;
}
+ public TermIndex getTermIndex() {
+ return TermIndex.newTermIndex(term, transactionIndex);
+ }
+
/**
* Generate String form of transaction info which need to be persisted in OM
* DB finally in byte array.
@@ -75,7 +80,7 @@ public final class OMTransactionInfo {
*/
private String generateTransactionInfo() {
StringBuilder stringBuilder = new StringBuilder();
- stringBuilder.append(currentTerm);
+ stringBuilder.append(term);
stringBuilder.append(TRANSACTION_INFO_SPLIT_KEY);
stringBuilder.append(transactionIndex);
@@ -109,13 +114,13 @@ public final class OMTransactionInfo {
return false;
}
OMTransactionInfo that = (OMTransactionInfo) o;
- return currentTerm == that.currentTerm &&
+ return term == that.term &&
transactionIndex == that.transactionIndex;
}
@Override
public int hashCode() {
- return Objects.hash(currentTerm, transactionIndex);
+ return Objects.hash(term, transactionIndex);
}
/**
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
index c042fcb..3f7429a 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
@@ -379,7 +379,7 @@ public class OzoneManagerStateMachine extends BaseStateMachine {
}
CompletableFuture<TermIndex> future = CompletableFuture.supplyAsync(
- () -> ozoneManager.installSnapshot(leaderNodeId),
+ () -> ozoneManager.installSnapshotFromLeader(leaderNodeId),
installSnapshotExecutor);
return future;
}
@@ -521,9 +521,9 @@ public class OzoneManagerStateMachine extends BaseStateMachine {
ozoneManager.getMetadataManager());
if (omTransactionInfo != null) {
setLastAppliedTermIndex(TermIndex.newTermIndex(
- omTransactionInfo.getCurrentTerm(),
+ omTransactionInfo.getTerm(),
omTransactionInfo.getTransactionIndex()));
- snapshotInfo.updateTermIndex(omTransactionInfo.getCurrentTerm(),
+ snapshotInfo.updateTermIndex(omTransactionInfo.getTerm(),
omTransactionInfo.getTransactionIndex());
}
LOG.info("LastAppliedIndex is set from TransactionInfo from OM DB as {}",
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
index 4aaaf13..ddb6841 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
@@ -228,15 +228,35 @@ public final class OzoneManagerRatisUtils {
}
/**
- * Obtain Transaction info from downloaded snapshot DB.
+ * Obtain OMTransactionInfo from Checkpoint.
+ */
+ public static OMTransactionInfo getTrxnInfoFromCheckpoint(
+ OzoneConfiguration conf, Path dbPath) throws Exception {
+
+ if (dbPath != null) {
+ Path dbDir = dbPath.getParent();
+ Path dbFile = dbPath.getFileName();
+ if (dbDir != null && dbFile != null) {
+ return getTransactionInfoFromDB(conf, dbDir, dbFile.toString());
+ }
+ }
+
+ throw new IOException("Checkpoint " + dbPath + " does not have proper " +
+ "DB location");
+ }
+
+ /**
+ * Obtain Transaction info from DB.
* @param tempConfig
+ * @param dbDir path to DB
* @return OMTransactionInfo
* @throws Exception
*/
- public static OMTransactionInfo getTransactionInfoFromDownloadedSnapshot(
- OzoneConfiguration tempConfig, Path dbDir) throws Exception {
- DBStore dbStore =
- OmMetadataManagerImpl.loadDB(tempConfig, dbDir.toFile());
+ private static OMTransactionInfo getTransactionInfoFromDB(
+ OzoneConfiguration tempConfig, Path dbDir, String dbName)
+ throws Exception {
+ DBStore dbStore = OmMetadataManagerImpl.loadDB(tempConfig, dbDir.toFile(),
+ dbName);
Table<String, OMTransactionInfo> transactionInfoTable =
dbStore.getTable(TRANSACTION_INFO_TABLE,
@@ -245,8 +265,11 @@ public final class OzoneManagerRatisUtils {
OMTransactionInfo omTransactionInfo =
transactionInfoTable.get(TRANSACTION_INFO_KEY);
dbStore.close();
- OzoneManager.LOG.info("Downloaded checkpoint with OMTransactionInfo {}",
- omTransactionInfo);
+
+ if (omTransactionInfo == null) {
+ throw new IOException("Failed to read OMTransactionInfo from DB " +
+ dbName + " at " + dbDir);
+ }
return omTransactionInfo;
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java
index 1c78251..a11c60b 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java
@@ -113,8 +113,10 @@ public class OzoneManagerSnapshotProvider {
public DBCheckpoint getOzoneManagerDBSnapshot(String leaderOMNodeID)
throws IOException {
String snapshotTime = Long.toString(System.currentTimeMillis());
- String snapshotFileName = Paths.get(omSnapshotDir.getAbsolutePath(),
- snapshotTime, OM_DB_NAME).toFile().getAbsolutePath();
+ String snapshotFileName = OM_DB_NAME + "-" + leaderOMNodeID
+ + "-" + snapshotTime;
+ String snapshotFilePath = Paths.get(omSnapshotDir.getAbsolutePath(),
+ snapshotFileName).toFile().getAbsolutePath();
File targetFile = new File(snapshotFileName + ".tar.gz");
String omCheckpointUrl = peerNodesMap.get(leaderOMNodeID)
@@ -141,11 +143,11 @@ public class OzoneManagerSnapshotProvider {
});
// Untar the checkpoint file.
- Path untarredDbDir = Paths.get(snapshotFileName);
+ Path untarredDbDir = Paths.get(snapshotFilePath);
FileUtil.unTar(targetFile, untarredDbDir.toFile());
FileUtils.deleteQuietly(targetFile);
- LOG.info("Sucessfully downloaded latest checkpoint from leader OM: {}",
+ LOG.info("Successfully downloaded latest checkpoint from leader OM: {}",
leaderOMNodeID);
RocksDBCheckpoint omCheckpoint = new RocksDBCheckpoint(untarredDbDir);
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java
index 054c97f..6226c5b 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java
@@ -76,7 +76,7 @@ public class TestOmMetadataManager {
OMTransactionInfo omTransactionInfo =
omMetadataManager.getTransactionInfoTable().get(TRANSACTION_INFO_KEY);
- Assert.assertEquals(3, omTransactionInfo.getCurrentTerm());
+ Assert.assertEquals(3, omTransactionInfo.getTerm());
Assert.assertEquals(250, omTransactionInfo.getTransactionIndex());
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithDummyResponse.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithDummyResponse.java
index 7b86006..372679b 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithDummyResponse.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithDummyResponse.java
@@ -139,7 +139,7 @@ public class TestOzoneManagerDoubleBufferWithDummyResponse {
Assert.assertEquals(lastAppliedIndex,
omTransactionInfo.getTransactionIndex());
- Assert.assertEquals(term, omTransactionInfo.getCurrentTerm());
+ Assert.assertEquals(term, omTransactionInfo.getTerm());
}
/**
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
index b369341..260e2cd 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
@@ -202,7 +202,7 @@ public class TestOzoneManagerDoubleBufferWithOMResponse {
Assert.assertEquals(lastAppliedIndex,
omTransactionInfo.getTransactionIndex());
- Assert.assertEquals(term, omTransactionInfo.getCurrentTerm());
+ Assert.assertEquals(term, omTransactionInfo.getTerm());
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 05/39: HDDS-3920. Too many redudant replications due to fail to get node's a… (#1163)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 16d8a3a5309ab97750e8ddec5ace7a727149ab32
Author: Sammi Chen <sa...@apache.org>
AuthorDate: Mon Jul 13 14:51:52 2020 +0800
HDDS-3920. Too many redudant replications due to fail to get node's a… (#1163)
(cherry picked from commit 9106f1421667c407dd5a533e1f1f37369424f0a3)
---
.../hadoop/hdds/protocol/DatanodeDetails.java | 1 +
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 2 +-
.../hdds/scm/container/ContainerReportHandler.java | 9 +++-
.../IncrementalContainerReportHandler.java | 13 +++--
.../hadoop/hdds/scm/node/SCMNodeManager.java | 63 ++++++++++++----------
.../TestIncrementalContainerReportHandler.java | 36 +++++++++++--
.../hdds/scm/node/TestNodeReportHandler.java | 5 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 2 +
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 5 +-
.../ozone/om/TestContainerReportWithKeys.java | 10 ++++
.../ReconIncrementalContainerReportHandler.java | 12 ++++-
...TestReconIncrementalContainerReportHandler.java | 30 ++++++++---
12 files changed, 140 insertions(+), 48 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
index 7fa7765..1b6a214 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
@@ -78,6 +78,7 @@ public class DatanodeDetails extends NodeImpl implements
this.hostName = datanodeDetails.hostName;
this.ports = datanodeDetails.ports;
this.setNetworkName(datanodeDetails.getNetworkName());
+ this.setParent(datanodeDetails.getParent());
}
/**
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java
index 08639ba..bf03174 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java
@@ -132,7 +132,7 @@ public class DeletedBlockLogImpl
if (block == null) {
// Should we make this an error ? How can we not find the deleted
// TXID?
- LOG.warn("Deleted TXID not found.");
+ LOG.warn("Deleted TXID {} not found.", txID);
continue;
}
DeletedBlocksTransaction.Builder builder = block.toBuilder();
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java
index 1b0b81f..8432e29 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java
@@ -102,8 +102,15 @@ public class ContainerReportHandler extends AbstractContainerReportHandler
public void onMessage(final ContainerReportFromDatanode reportFromDatanode,
final EventPublisher publisher) {
- final DatanodeDetails datanodeDetails =
+ final DatanodeDetails dnFromReport =
reportFromDatanode.getDatanodeDetails();
+ DatanodeDetails datanodeDetails =
+ nodeManager.getNodeByUuid(dnFromReport.getUuidString());
+ if (datanodeDetails == null) {
+ LOG.warn("Received container report from unknown datanode {}",
+ dnFromReport);
+ return;
+ }
final ContainerReportsProto containerReport =
reportFromDatanode.getReport();
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java
index 017cc5c..c2148df 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java
@@ -54,16 +54,23 @@ public class IncrementalContainerReportHandler extends
@Override
public void onMessage(final IncrementalContainerReportFromDatanode report,
final EventPublisher publisher) {
+ final DatanodeDetails dnFromReport = report.getDatanodeDetails();
if (LOG.isDebugEnabled()) {
LOG.debug("Processing incremental container report from data node {}",
- report.getDatanodeDetails().getUuid());
+ dnFromReport.getUuid());
+ }
+ DatanodeDetails dd =
+ nodeManager.getNodeByUuid(dnFromReport.getUuidString());
+ if (dd == null) {
+ LOG.warn("Received container report from unknown datanode {}",
+ dnFromReport);
+ return;
}
boolean success = true;
for (ContainerReplicaProto replicaProto :
report.getReport().getReportList()) {
try {
- final DatanodeDetails dd = report.getDatanodeDetails();
final ContainerID id = ContainerID.valueof(
replicaProto.getContainerID());
if (!replicaProto.getState().equals(
@@ -81,7 +88,7 @@ public class IncrementalContainerReportHandler extends
} catch (IOException e) {
success = false;
LOG.error("Exception while processing ICR for container {}",
- replicaProto.getContainerID());
+ replicaProto.getContainerID(), e);
}
}
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
index 5a3851a..005881c 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
@@ -241,34 +241,43 @@ public class SCMNodeManager implements NodeManager {
DatanodeDetails datanodeDetails, NodeReportProto nodeReport,
PipelineReportsProto pipelineReportsProto) {
- InetAddress dnAddress = Server.getRemoteIp();
- if (dnAddress != null) {
- // Mostly called inside an RPC, update ip and peer hostname
- datanodeDetails.setHostName(dnAddress.getHostName());
- datanodeDetails.setIpAddress(dnAddress.getHostAddress());
- }
- try {
- String dnsName;
- String networkLocation;
- datanodeDetails.setNetworkName(datanodeDetails.getUuidString());
- if (useHostname) {
- dnsName = datanodeDetails.getHostName();
- } else {
- dnsName = datanodeDetails.getIpAddress();
- }
- networkLocation = nodeResolve(dnsName);
- if (networkLocation != null) {
- datanodeDetails.setNetworkLocation(networkLocation);
+ if (!isNodeRegistered(datanodeDetails)) {
+ InetAddress dnAddress = Server.getRemoteIp();
+ if (dnAddress != null) {
+ // Mostly called inside an RPC, update ip and peer hostname
+ datanodeDetails.setHostName(dnAddress.getHostName());
+ datanodeDetails.setIpAddress(dnAddress.getHostAddress());
}
- nodeStateManager.addNode(datanodeDetails);
- clusterMap.add(datanodeDetails);
- addEntryTodnsToUuidMap(dnsName, datanodeDetails.getUuidString());
- // Updating Node Report, as registration is successful
- processNodeReport(datanodeDetails, nodeReport);
- LOG.info("Registered Data node : {}", datanodeDetails);
- } catch (NodeAlreadyExistsException e) {
- if (LOG.isTraceEnabled()) {
- LOG.trace("Datanode is already registered. Datanode: {}",
+ try {
+ String dnsName;
+ String networkLocation;
+ datanodeDetails.setNetworkName(datanodeDetails.getUuidString());
+ if (useHostname) {
+ dnsName = datanodeDetails.getHostName();
+ } else {
+ dnsName = datanodeDetails.getIpAddress();
+ }
+ networkLocation = nodeResolve(dnsName);
+ if (networkLocation != null) {
+ datanodeDetails.setNetworkLocation(networkLocation);
+ }
+
+ clusterMap.add(datanodeDetails);
+ nodeStateManager.addNode(datanodeDetails);
+ // Check that datanode in nodeStateManager has topology parent set
+ DatanodeDetails dn = nodeStateManager.getNode(datanodeDetails);
+ Preconditions.checkState(dn.getParent() != null);
+ addEntryTodnsToUuidMap(dnsName, datanodeDetails.getUuidString());
+ // Updating Node Report, as registration is successful
+ processNodeReport(datanodeDetails, nodeReport);
+ LOG.info("Registered Data node : {}", datanodeDetails);
+ } catch (NodeAlreadyExistsException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Datanode is already registered. Datanode: {}",
+ datanodeDetails.toString());
+ }
+ } catch (NodeNotFoundException e) {
+ LOG.error("Cannot find datanode {} from nodeStateManager",
datanodeDetails.toString());
}
}
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestIncrementalContainerReportHandler.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestIncrementalContainerReportHandler.java
index efa333d..1af2f73 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestIncrementalContainerReportHandler.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestIncrementalContainerReportHandler.java
@@ -17,7 +17,7 @@
*/
package org.apache.hadoop.hdds.scm.container;
-import org.apache.hadoop.hdds.conf.ConfigurationSource;
+import org.apache.hadoop.hdds.HddsConfigKeys;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
@@ -26,10 +26,16 @@ import org.apache.hadoop.hdds.protocol.proto
.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto;
import org.apache.hadoop.hdds.protocol.proto
.StorageContainerDatanodeProtocolProtos.IncrementalContainerReportProto;
+import org.apache.hadoop.hdds.scm.net.NetworkTopology;
+import org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl;
import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.apache.hadoop.hdds.scm.node.SCMNodeManager;
import org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher
.IncrementalContainerReportFromDatanode;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
import org.apache.hadoop.hdds.server.events.EventPublisher;
+import org.apache.hadoop.hdds.server.events.EventQueue;
+import org.apache.hadoop.test.GenericTestUtils;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -37,7 +43,10 @@ import org.junit.Test;
import org.mockito.Mockito;
import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
import java.util.Set;
+import java.util.UUID;
import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
import static org.apache.hadoop.hdds.scm.TestUtils.getContainer;
@@ -55,9 +64,18 @@ public class TestIncrementalContainerReportHandler {
@Before
public void setup() throws IOException {
- final ConfigurationSource conf = new OzoneConfiguration();
+ final OzoneConfiguration conf = new OzoneConfiguration();
+ final String path =
+ GenericTestUtils.getTempPath(UUID.randomUUID().toString());
+ Path scmPath = Paths.get(path, "scm-meta");
+ conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, scmPath.toString());
this.containerManager = Mockito.mock(ContainerManager.class);
- this.nodeManager = Mockito.mock(NodeManager.class);
+ NetworkTopology clusterMap = new NetworkTopologyImpl(conf);
+ EventQueue eventQueue = new EventQueue();
+ SCMStorageConfig storageConfig = new SCMStorageConfig(conf);
+ this.nodeManager =
+ new SCMNodeManager(conf, storageConfig, eventQueue, clusterMap);
+
this.containerStateManager = new ContainerStateManager(conf);
this.publisher = Mockito.mock(EventPublisher.class);
@@ -105,6 +123,9 @@ public class TestIncrementalContainerReportHandler {
final DatanodeDetails datanodeOne = randomDatanodeDetails();
final DatanodeDetails datanodeTwo = randomDatanodeDetails();
final DatanodeDetails datanodeThree = randomDatanodeDetails();
+ nodeManager.register(datanodeOne, null, null);
+ nodeManager.register(datanodeTwo, null, null);
+ nodeManager.register(datanodeThree, null, null);
final Set<ContainerReplica> containerReplicas = getReplicas(
container.containerID(),
ContainerReplicaProto.State.CLOSING,
@@ -139,6 +160,9 @@ public class TestIncrementalContainerReportHandler {
final DatanodeDetails datanodeOne = randomDatanodeDetails();
final DatanodeDetails datanodeTwo = randomDatanodeDetails();
final DatanodeDetails datanodeThree = randomDatanodeDetails();
+ nodeManager.register(datanodeOne, null, null);
+ nodeManager.register(datanodeTwo, null, null);
+ nodeManager.register(datanodeThree, null, null);
final Set<ContainerReplica> containerReplicas = getReplicas(
container.containerID(),
ContainerReplicaProto.State.CLOSING,
@@ -174,6 +198,9 @@ public class TestIncrementalContainerReportHandler {
final DatanodeDetails datanodeOne = randomDatanodeDetails();
final DatanodeDetails datanodeTwo = randomDatanodeDetails();
final DatanodeDetails datanodeThree = randomDatanodeDetails();
+ nodeManager.register(datanodeOne, null, null);
+ nodeManager.register(datanodeTwo, null, null);
+ nodeManager.register(datanodeThree, null, null);
final Set<ContainerReplica> containerReplicas = getReplicas(
container.containerID(),
ContainerReplicaProto.State.CLOSING,
@@ -212,6 +239,9 @@ public class TestIncrementalContainerReportHandler {
final DatanodeDetails datanodeOne = randomDatanodeDetails();
final DatanodeDetails datanodeTwo = randomDatanodeDetails();
final DatanodeDetails datanodeThree = randomDatanodeDetails();
+ nodeManager.register(datanodeOne, null, null);
+ nodeManager.register(datanodeTwo, null, null);
+ nodeManager.register(datanodeThree, null, null);
final Set<ContainerReplica> containerReplicas = getReplicas(
container.containerID(),
ContainerReplicaProto.State.CLOSED,
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeReportHandler.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeReportHandler.java
index 7b4d841..69b031c 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeReportHandler.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeReportHandler.java
@@ -26,6 +26,7 @@ import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolPro
import org.apache.hadoop.hdds.scm.TestUtils;
import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeMetric;
import org.apache.hadoop.hdds.scm.net.NetworkTopology;
+import org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl;
import org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.NodeReportFromDatanode;
import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
import org.apache.hadoop.hdds.server.events.Event;
@@ -56,9 +57,9 @@ public class TestNodeReportHandler implements EventPublisher {
OzoneConfiguration conf = new OzoneConfiguration();
SCMStorageConfig storageConfig = Mockito.mock(SCMStorageConfig.class);
Mockito.when(storageConfig.getClusterID()).thenReturn("cluster1");
+ NetworkTopology clusterMap = new NetworkTopologyImpl(conf);
nodeManager =
- new SCMNodeManager(conf, storageConfig, new EventQueue(), Mockito.mock(
- NetworkTopology.class));
+ new SCMNodeManager(conf, storageConfig, new EventQueue(), clusterMap);
nodeReportHandler = new NodeReportHandler(nodeManager);
}
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
index b167a38..df5cb2d 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
@@ -1133,6 +1133,8 @@ public class TestSCMNodeManager {
List<DatanodeDetails> nodeList = nodeManager.getAllNodes();
nodeList.stream().forEach(node ->
Assert.assertTrue(node.getNetworkLocation().startsWith("/rack1/ng")));
+ nodeList.stream().forEach(node ->
+ Assert.assertTrue(node.getParent() != null));
}
}
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodeMetrics.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodeMetrics.java
index 3625e34..7576e8b 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodeMetrics.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodeMetrics.java
@@ -30,7 +30,7 @@ import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolPro
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReportsProto;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.StorageReportProto;
import org.apache.hadoop.hdds.scm.TestUtils;
-import org.apache.hadoop.hdds.scm.net.NetworkTopology;
+import org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl;
import org.apache.hadoop.hdds.scm.node.SCMNodeManager;
import org.apache.hadoop.hdds.scm.node.SCMNodeMetrics;
import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
@@ -45,7 +45,6 @@ import org.junit.Assert;
import static org.junit.Assert.assertEquals;
import org.junit.BeforeClass;
import org.junit.Test;
-import org.mockito.Mockito;
/**
* Test cases to verify the metrics exposed by SCMNodeManager.
@@ -64,7 +63,7 @@ public class TestSCMNodeMetrics {
SCMStorageConfig config =
new SCMStorageConfig(NodeType.DATANODE, new File("/tmp"), "storage");
nodeManager = new SCMNodeManager(source, config, publisher,
- Mockito.mock(NetworkTopology.class));
+ new NetworkTopologyImpl(source));
registeredDatanode = DatanodeDetails.newBuilder()
.setHostName("localhost")
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestContainerReportWithKeys.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestContainerReportWithKeys.java
index 8984e76..7f049a3 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestContainerReportWithKeys.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestContainerReportWithKeys.java
@@ -22,7 +22,9 @@ import org.apache.hadoop.hdds.client.ReplicationFactor;
import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.apache.hadoop.ozone.client.*;
import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
@@ -32,6 +34,7 @@ import org.apache.hadoop.ozone.om.helpers.OmKeyArgs;
import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
import org.apache.hadoop.hdds.scm.server.StorageContainerManager;
import org.junit.AfterClass;
+import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -42,6 +45,7 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.HashMap;
+import java.util.Set;
/**
* This class tests container report with DN container state info.
@@ -122,6 +126,12 @@ public class TestContainerReportWithKeys {
ContainerInfo cinfo = scm.getContainerInfo(keyInfo.getContainerID());
+ Set<ContainerReplica> replicas =
+ scm.getContainerManager().getContainerReplicas(
+ new ContainerID(keyInfo.getContainerID()));
+ Assert.assertTrue(replicas.size() == 1);
+ replicas.stream().forEach(rp ->
+ Assert.assertTrue(rp.getDatanodeDetails().getParent() != null));
LOG.info("SCM Container Info keyCount: {} usedBytes: {}",
cinfo.getNumberOfKeys(), cinfo.getUsedBytes());
diff --git a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconIncrementalContainerReportHandler.java b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconIncrementalContainerReportHandler.java
index a5d946e..b538caf 100644
--- a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconIncrementalContainerReportHandler.java
+++ b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconIncrementalContainerReportHandler.java
@@ -50,9 +50,18 @@ public class ReconIncrementalContainerReportHandler
@Override
public void onMessage(final IncrementalContainerReportFromDatanode report,
final EventPublisher publisher) {
+ final DatanodeDetails dnFromReport = report.getDatanodeDetails();
if (LOG.isDebugEnabled()) {
LOG.debug("Processing incremental container report from data node {}",
- report.getDatanodeDetails());
+ dnFromReport);
+ }
+
+ DatanodeDetails dd =
+ getNodeManager().getNodeByUuid(dnFromReport.getUuidString());
+ if (dd == null) {
+ LOG.warn("Received container report from unknown datanode {}",
+ dnFromReport);
+ return;
}
ReconContainerManager containerManager =
@@ -61,7 +70,6 @@ public class ReconIncrementalContainerReportHandler
for (ContainerReplicaProto replicaProto :
report.getReport().getReportList()) {
try {
- final DatanodeDetails dd = report.getDatanodeDetails();
final ContainerID id = ContainerID.valueof(
replicaProto.getContainerID());
try {
diff --git a/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconIncrementalContainerReportHandler.java b/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconIncrementalContainerReportHandler.java
index d4f28c0..dacf293 100644
--- a/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconIncrementalContainerReportHandler.java
+++ b/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconIncrementalContainerReportHandler.java
@@ -22,21 +22,30 @@ import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanode
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto.State;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.IncrementalContainerReportProto;
import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.net.NetworkTopology;
+import org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl;
import org.apache.hadoop.hdds.scm.node.NodeManager;
+import org.apache.hadoop.hdds.scm.node.SCMNodeManager;
import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
import org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.IncrementalContainerReportFromDatanode;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
import org.apache.hadoop.hdds.server.events.EventPublisher;
+import org.apache.hadoop.hdds.server.events.EventQueue;
+import org.apache.hadoop.test.GenericTestUtils;
import org.junit.Test;
/**
@@ -59,17 +68,26 @@ public class TestReconIncrementalContainerReportHandler
datanodeDetails.getUuidString());
when(reportMock.getReport()).thenReturn(containerReport);
- NodeManager nodeManagerMock = mock(NodeManager.class);
+ final String path =
+ GenericTestUtils.getTempPath(UUID.randomUUID().toString());
+ Path scmPath = Paths.get(path, "scm-meta");
+ final OzoneConfiguration conf = new OzoneConfiguration();
+ conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, scmPath.toString());
+ NetworkTopology clusterMap = new NetworkTopologyImpl(conf);
+ EventQueue eventQueue = new EventQueue();
+ SCMStorageConfig storageConfig = new SCMStorageConfig(conf);
+ NodeManager nodeManager =
+ new SCMNodeManager(conf, storageConfig, eventQueue, clusterMap);
+ nodeManager.register(datanodeDetails, null, null);
ReconContainerManager containerManager = getContainerManager();
ReconIncrementalContainerReportHandler reconIcr =
- new ReconIncrementalContainerReportHandler(nodeManagerMock,
+ new ReconIncrementalContainerReportHandler(nodeManager,
containerManager);
EventPublisher eventPublisherMock = mock(EventPublisher.class);
reconIcr.onMessage(reportMock, eventPublisherMock);
- verify(nodeManagerMock, times(1))
- .addContainer(datanodeDetails, containerID);
+ nodeManager.addContainer(datanodeDetails, containerID);
assertTrue(containerManager.exists(containerID));
assertEquals(1, containerManager.getContainerReplicas(containerID).size());
}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 25/39: HDDS-3612. Allow mounting bucket under other
volume (#1104)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 6b666bfdc23a8e7354bd05e93872480a48fcaa2d
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Fri Jul 17 06:02:31 2020 +0200
HDDS-3612. Allow mounting bucket under other volume (#1104)
(cherry picked from commit 7aff2f05271e1d421bc48d1a02c1b7ad596888cd)
---
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../org/apache/hadoop/ozone/client/BucketArgs.java | 33 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 23 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 11 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 5 -
.../hadoop/ozone/om/exceptions/OMException.java | 4 +-
.../ozone/om/helpers/BucketEncryptionKeyInfo.java | 4 +
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 142 +++++++--
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 18 ++
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 22 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 18 +-
.../dist/src/main/compose/ozonesecure/test.sh | 13 +-
.../dist/src/main/smoketest/basic/links.robot | 152 ++++++++++
.../dist/src/main/smoketest/commonlib.robot | 34 +--
.../smoketest/{commonlib.robot => lib/os.robot} | 36 +--
.../{s3/bucketdelete.robot => lib/os_tests.robot} | 35 +--
.../dist/src/main/smoketest/ozone-lib/shell.robot | 48 +++
.../src/main/smoketest/ozone-lib/shell_tests.robot | 58 ++++
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 2 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 16 +-
hadoop-ozone/dist/src/main/smoketest/robot.robot | 81 +++++
.../src/main/smoketest/s3/MultipartUpload.robot | 7 +-
.../dist/src/main/smoketest/s3/bucketdelete.robot | 12 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 33 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 100 +++---
.../src/main/proto/OmClientProtocol.proto | 4 +
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 112 ++++---
.../org/apache/hadoop/ozone/om/OzoneManager.java | 334 +++++++++++++++------
.../org/apache/hadoop/ozone/om/ResolvedBucket.java | 111 +++++++
.../om/request/bucket/OMBucketCreateRequest.java | 14 +
.../om/request/file/OMDirectoryCreateRequest.java | 4 +
.../ozone/om/request/file/OMFileCreateRequest.java | 6 +-
.../om/request/key/OMAllocateBlockRequest.java | 9 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 27 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 8 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 15 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 19 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 11 +
.../ozone/om/request/key/OMKeysDeleteRequest.java | 48 ++-
.../om/request/key/OMTrashRecoverRequest.java | 7 +
.../S3InitiateMultipartUploadRequest.java | 33 +-
.../multipart/S3MultipartUploadAbortRequest.java | 25 +-
.../S3MultipartUploadCommitPartRequest.java | 42 +--
.../S3MultipartUploadCompleteRequest.java | 71 +++--
.../request/file/TestOMDirectoryCreateRequest.java | 4 +
.../ozone/om/request/key/TestOMKeyRequest.java | 8 +
.../s3/multipart/TestS3MultipartRequest.java | 10 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 4 +-
.../hadoop/ozone/shell/bucket/BucketCommands.java | 1 +
.../ozone/shell/bucket/LinkBucketHandler.java | 79 +++++
50 files changed, 1448 insertions(+), 467 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
index a473948..e340b32 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
@@ -298,6 +298,8 @@ public final class OzoneConsts {
public static final String BUCKET_ENCRYPTION_KEY = "bucketEncryptionKey";
public static final String DELETED_KEYS_LIST = "deletedKeysList";
public static final String UNDELETED_KEYS_LIST = "unDeletedKeysList";
+ public static final String SOURCE_VOLUME = "sourceVolume";
+ public static final String SOURCE_BUCKET = "sourceBucket";
diff --git a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BucketArgs.java b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BucketArgs.java
index 5bae15d..6c5d1dd 100644
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BucketArgs.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BucketArgs.java
@@ -54,6 +54,8 @@ public final class BucketArgs {
* Bucket encryption key name.
*/
private String bucketEncryptionKey;
+ private final String sourceVolume;
+ private final String sourceBucket;
/**
* Private constructor, constructed via builder.
@@ -62,15 +64,19 @@ public final class BucketArgs {
* @param acls list of ACLs.
* @param metadata map of bucket metadata
* @param bucketEncryptionKey bucket encryption key name
+ * @param sourceVolume
+ * @param sourceBucket
*/
private BucketArgs(Boolean versioning, StorageType storageType,
- List<OzoneAcl> acls, Map<String, String> metadata,
- String bucketEncryptionKey) {
+ List<OzoneAcl> acls, Map<String, String> metadata,
+ String bucketEncryptionKey, String sourceVolume, String sourceBucket) {
this.acls = acls;
this.versioning = versioning;
this.storageType = storageType;
this.metadata = metadata;
this.bucketEncryptionKey = bucketEncryptionKey;
+ this.sourceVolume = sourceVolume;
+ this.sourceBucket = sourceBucket;
}
/**
@@ -123,6 +129,14 @@ public final class BucketArgs {
return new BucketArgs.Builder();
}
+ public String getSourceVolume() {
+ return sourceVolume;
+ }
+
+ public String getSourceBucket() {
+ return sourceBucket;
+ }
+
/**
* Builder for OmBucketInfo.
*/
@@ -132,6 +146,8 @@ public final class BucketArgs {
private List<OzoneAcl> acls;
private Map<String, String> metadata;
private String bucketEncryptionKey;
+ private String sourceVolume;
+ private String sourceBucket;
public Builder() {
metadata = new HashMap<>();
@@ -161,13 +177,24 @@ public final class BucketArgs {
this.bucketEncryptionKey = bek;
return this;
}
+
+ public BucketArgs.Builder setSourceVolume(String volume) {
+ sourceVolume = volume;
+ return this;
+ }
+
+ public BucketArgs.Builder setSourceBucket(String bucket) {
+ sourceBucket = bucket;
+ return this;
+ }
+
/**
* Constructs the BucketArgs.
* @return instance of BucketArgs.
*/
public BucketArgs build() {
return new BucketArgs(versioning, storageType, acls, metadata,
- bucketEncryptionKey);
+ bucketEncryptionKey, sourceVolume, sourceBucket);
}
}
}
diff --git a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
index d22b846..79712bb 100644
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
@@ -109,6 +109,8 @@ public class OzoneBucket extends WithMetadata {
private OzoneObj ozoneObj;
+ private String sourceVolume;
+ private String sourceBucket;
private OzoneBucket(ConfigurationSource conf, String volumeName,
String bucketName, ReplicationFactor defaultReplication,
@@ -138,11 +140,13 @@ public class OzoneBucket extends WithMetadata {
.setResType(OzoneObj.ResourceType.BUCKET)
.setStoreType(OzoneObj.StoreType.OZONE).build();
}
+
@SuppressWarnings("parameternumber")
public OzoneBucket(ConfigurationSource conf, ClientProtocol proxy,
String volumeName, String bucketName, StorageType storageType,
Boolean versioning, long creationTime, Map<String, String> metadata,
- String encryptionKeyName) {
+ String encryptionKeyName,
+ String sourceVolume, String sourceBucket) {
this(conf, volumeName, bucketName, null, null, proxy);
this.storageType = storageType;
this.versioning = versioning;
@@ -150,6 +154,8 @@ public class OzoneBucket extends WithMetadata {
this.creationTime = Instant.ofEpochMilli(creationTime);
this.metadata = metadata;
this.encryptionKeyName = encryptionKeyName;
+ this.sourceVolume = sourceVolume;
+ this.sourceBucket = sourceBucket;
modificationTime = Instant.now();
if (modificationTime.isBefore(this.creationTime)) {
modificationTime = Instant.ofEpochSecond(
@@ -161,9 +167,10 @@ public class OzoneBucket extends WithMetadata {
public OzoneBucket(ConfigurationSource conf, ClientProtocol proxy,
String volumeName, String bucketName, StorageType storageType,
Boolean versioning, long creationTime, long modificationTime,
- Map<String, String> metadata, String encryptionKeyName) {
+ Map<String, String> metadata, String encryptionKeyName,
+ String sourceVolume, String sourceBucket) {
this(conf, proxy, volumeName, bucketName, storageType, versioning,
- creationTime, metadata, encryptionKeyName);
+ creationTime, metadata, encryptionKeyName, sourceVolume, sourceBucket);
this.modificationTime = Instant.ofEpochMilli(modificationTime);
}
@@ -306,6 +313,16 @@ public class OzoneBucket extends WithMetadata {
return encryptionKeyName;
}
+ public String getSourceVolume() {
+ return sourceVolume;
+ }
+
+ public String getSourceBucket() {
+ return sourceBucket;
+ }
+
+ /**
+ * Builder for OmBucketInfo.
/**
* Adds ACLs to the Bucket.
* @param addAcl ACL to be added
diff --git a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
index bfc2f0d..dc37f09 100644
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
@@ -448,6 +448,8 @@ public class RpcClient implements ClientProtocol {
.setIsVersionEnabled(isVersionEnabled)
.addAllMetadata(bucketArgs.getMetadata())
.setStorageType(storageType)
+ .setSourceVolume(bucketArgs.getSourceVolume())
+ .setSourceBucket(bucketArgs.getSourceBucket())
.setAcls(listOfAcls.stream().distinct().collect(Collectors.toList()));
if (bek != null) {
@@ -614,7 +616,10 @@ public class RpcClient implements ClientProtocol {
bucketInfo.getModificationTime(),
bucketInfo.getMetadata(),
bucketInfo.getEncryptionKeyInfo() != null ? bucketInfo
- .getEncryptionKeyInfo().getKeyName() : null);
+ .getEncryptionKeyInfo().getKeyName() : null,
+ bucketInfo.getSourceVolume(),
+ bucketInfo.getSourceBucket()
+ );
}
@Override
@@ -635,7 +640,9 @@ public class RpcClient implements ClientProtocol {
bucket.getModificationTime(),
bucket.getMetadata(),
bucket.getEncryptionKeyInfo() != null ? bucket
- .getEncryptionKeyInfo().getKeyName() : null))
+ .getEncryptionKeyInfo().getKeyName() : null,
+ bucket.getSourceVolume(),
+ bucket.getSourceBucket()))
.collect(Collectors.toList());
}
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
index 31cccac..6b34e81 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
@@ -40,10 +40,6 @@ public enum OMAction implements AuditAction {
PURGE_KEYS,
DELETE_KEYS,
- // S3 Bucket
- CREATE_S3_BUCKET,
- DELETE_S3_BUCKET,
-
// READ Actions
CHECK_VOLUME_ACCESS,
LIST_BUCKETS,
@@ -53,7 +49,6 @@ public enum OMAction implements AuditAction {
READ_VOLUME,
READ_BUCKET,
READ_KEY,
- LIST_S3BUCKETS,
INITIATE_MULTIPART_UPLOAD,
COMMIT_MULTIPART_UPLOAD_PARTKEY,
COMPLETE_MULTIPART_UPLOAD,
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
index e2b3418..bab8d94 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
@@ -223,6 +223,8 @@ public class OMException extends IOException {
INVALID_VOLUME_NAME,
- PARTIAL_DELETE
+ PARTIAL_DELETE,
+
+ DETECTED_LOOP_IN_BUCKET_LINKS,
}
}
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/BucketEncryptionKeyInfo.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/BucketEncryptionKeyInfo.java
index e1ae0bb..c180138 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/BucketEncryptionKeyInfo.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/BucketEncryptionKeyInfo.java
@@ -49,6 +49,10 @@ public class BucketEncryptionKeyInfo {
return version;
}
+ public BucketEncryptionKeyInfo copy() {
+ return new BucketEncryptionKeyInfo(version, suite, keyName);
+ }
+
/**
* Builder for BucketEncryptionKeyInfo.
*/
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java
index e9a8cbc..abbe395 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java
@@ -76,6 +76,10 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
*/
private BucketEncryptionKeyInfo bekInfo;
+ private final String sourceVolume;
+
+ private final String sourceBucket;
+
/**
* Private constructor, constructed via builder.
* @param volumeName - Volume name.
@@ -87,19 +91,23 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
* @param modificationTime - Bucket modification time.
* @param metadata - metadata.
* @param bekInfo - bucket encryption key info.
+ * @param sourceVolume - source volume for bucket links, null otherwise
+ * @param sourceBucket - source bucket for bucket links, null otherwise
*/
@SuppressWarnings("checkstyle:ParameterNumber")
private OmBucketInfo(String volumeName,
- String bucketName,
- List<OzoneAcl> acls,
- boolean isVersionEnabled,
- StorageType storageType,
- long creationTime,
- long modificationTime,
- long objectID,
- long updateID,
- Map<String, String> metadata,
- BucketEncryptionKeyInfo bekInfo) {
+ String bucketName,
+ List<OzoneAcl> acls,
+ boolean isVersionEnabled,
+ StorageType storageType,
+ long creationTime,
+ long modificationTime,
+ long objectID,
+ long updateID,
+ Map<String, String> metadata,
+ BucketEncryptionKeyInfo bekInfo,
+ String sourceVolume,
+ String sourceBucket) {
this.volumeName = volumeName;
this.bucketName = bucketName;
this.acls = acls;
@@ -111,6 +119,8 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
this.updateID = updateID;
this.metadata = metadata;
this.bekInfo = bekInfo;
+ this.sourceVolume = sourceVolume;
+ this.sourceBucket = sourceBucket;
}
/**
@@ -208,6 +218,18 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
return bekInfo;
}
+ public String getSourceVolume() {
+ return sourceVolume;
+ }
+
+ public String getSourceBucket() {
+ return sourceBucket;
+ }
+
+ public boolean isLink() {
+ return sourceVolume != null && sourceBucket != null;
+ }
+
/**
* Returns new builder class that builds a OmBucketInfo.
*
@@ -235,6 +257,10 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
(bekInfo != null) ? bekInfo.getKeyName() : null);
auditMap.put(OzoneConsts.MODIFICATION_TIME,
String.valueOf(this.modificationTime));
+ if (isLink()) {
+ auditMap.put(OzoneConsts.SOURCE_VOLUME, sourceVolume);
+ auditMap.put(OzoneConsts.SOURCE_BUCKET, sourceBucket);
+ }
return auditMap;
}
@@ -242,7 +268,22 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
* Return a new copy of the object.
*/
public OmBucketInfo copyObject() {
- OmBucketInfo.Builder builder = new OmBucketInfo.Builder()
+ Builder builder = toBuilder();
+
+ if (bekInfo != null) {
+ builder.setBucketEncryptionKey(bekInfo.copy());
+ }
+
+ builder.acls.clear();
+ acls.forEach(acl -> builder.addAcl(new OzoneAcl(acl.getType(),
+ acl.getName(), (BitSet) acl.getAclBitSet().clone(),
+ acl.getAclScope())));
+
+ return builder.build();
+ }
+
+ public Builder toBuilder() {
+ return new Builder()
.setVolumeName(volumeName)
.setBucketName(bucketName)
.setStorageType(storageType)
@@ -251,19 +292,11 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
.setModificationTime(modificationTime)
.setObjectID(objectID)
.setUpdateID(updateID)
- .setBucketEncryptionKey(bekInfo != null ?
- new BucketEncryptionKeyInfo(bekInfo.getVersion(),
- bekInfo.getSuite(), bekInfo.getKeyName()) : null);
-
- acls.forEach(acl -> builder.addAcl(new OzoneAcl(acl.getType(),
- acl.getName(), (BitSet) acl.getAclBitSet().clone(),
- acl.getAclScope())));
-
- if (metadata != null) {
- metadata.forEach((k, v) -> builder.addMetadata(k, v));
- }
- return builder.build();
-
+ .setBucketEncryptionKey(bekInfo)
+ .setSourceVolume(sourceVolume)
+ .setSourceBucket(sourceBucket)
+ .setAcls(acls)
+ .addAllMetadata(metadata);
}
/**
@@ -281,6 +314,8 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
private long updateID;
private Map<String, String> metadata;
private BucketEncryptionKeyInfo bekInfo;
+ private String sourceVolume;
+ private String sourceBucket;
public Builder() {
//Default values
@@ -362,6 +397,16 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
return this;
}
+ public Builder setSourceVolume(String volume) {
+ this.sourceVolume = volume;
+ return this;
+ }
+
+ public Builder setSourceBucket(String bucket) {
+ this.sourceBucket = bucket;
+ return this;
+ }
+
/**
* Constructs the OmBucketInfo.
* @return instance of OmBucketInfo.
@@ -375,7 +420,7 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
return new OmBucketInfo(volumeName, bucketName, acls, isVersionEnabled,
storageType, creationTime, modificationTime, objectID, updateID,
- metadata, bekInfo);
+ metadata, bekInfo, sourceVolume, sourceBucket);
}
}
@@ -397,6 +442,12 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
if (bekInfo != null && bekInfo.getKeyName() != null) {
bib.setBeinfo(OMPBHelper.convert(bekInfo));
}
+ if (sourceVolume != null) {
+ bib.setSourceVolume(sourceVolume);
+ }
+ if (sourceBucket != null) {
+ bib.setSourceBucket(sourceBucket);
+ }
return bib.build();
}
@@ -428,17 +479,28 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
if (bucketInfo.hasBeinfo()) {
obib.setBucketEncryptionKey(OMPBHelper.convert(bucketInfo.getBeinfo()));
}
+ if (bucketInfo.hasSourceVolume()) {
+ obib.setSourceVolume(bucketInfo.getSourceVolume());
+ }
+ if (bucketInfo.hasSourceBucket()) {
+ obib.setSourceBucket(bucketInfo.getSourceBucket());
+ }
return obib.build();
}
@Override
public String getObjectInfo() {
+ String sourceInfo = sourceVolume != null && sourceBucket != null
+ ? ", source='" + sourceVolume + "/" + sourceBucket + "'"
+ : "";
+
return "OMBucketInfo{" +
- "volume='" + volumeName + '\'' +
- ", bucket='" + bucketName + '\'' +
- ", isVersionEnabled='" + isVersionEnabled + '\'' +
- ", storageType='" + storageType + '\'' +
- ", creationTime='" + creationTime + '\'' +
+ "volume='" + volumeName + "'" +
+ ", bucket='" + bucketName + "'" +
+ ", isVersionEnabled='" + isVersionEnabled + "'" +
+ ", storageType='" + storageType + "'" +
+ ", creationTime='" + creationTime + "'" +
+ sourceInfo +
'}';
}
@@ -460,6 +522,8 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
storageType == that.storageType &&
objectID == that.objectID &&
updateID == that.updateID &&
+ Objects.equals(sourceVolume, that.sourceVolume) &&
+ Objects.equals(sourceBucket, that.sourceBucket) &&
Objects.equals(metadata, that.metadata) &&
Objects.equals(bekInfo, that.bekInfo);
}
@@ -468,4 +532,22 @@ public final class OmBucketInfo extends WithObjectID implements Auditable {
public int hashCode() {
return Objects.hash(volumeName, bucketName);
}
+
+ @Override
+ public String toString() {
+ return "OmBucketInfo{" +
+ "volumeName='" + volumeName + "'" +
+ ", bucketName='" + bucketName + "'" +
+ ", acls=" + acls +
+ ", isVersionEnabled=" + isVersionEnabled +
+ ", storageType=" + storageType +
+ ", creationTime=" + creationTime +
+ ", bekInfo=" + bekInfo +
+ ", sourceVolume='" + sourceVolume + "'" +
+ ", sourceBucket='" + sourceBucket + "'" +
+ ", objectID=" + objectID +
+ ", updateID=" + updateID +
+ ", metadata=" + metadata +
+ '}';
+ }
}
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java
index 2a882a4..c08c988 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java
@@ -162,6 +162,24 @@ public final class OmKeyArgs implements Auditable {
locationInfoList.add(locationInfo);
}
+ public OmKeyArgs.Builder toBuilder() {
+ return new OmKeyArgs.Builder()
+ .setVolumeName(volumeName)
+ .setBucketName(bucketName)
+ .setKeyName(keyName)
+ .setDataSize(dataSize)
+ .setType(type)
+ .setFactor(factor)
+ .setLocationInfoList(locationInfoList)
+ .setIsMultipartKey(isMultipartKey)
+ .setMultipartUploadID(multipartUploadID)
+ .setMultipartUploadPartNumber(multipartUploadPartNumber)
+ .addAllMetadata(metadata)
+ .setRefreshPipeline(refreshPipeline)
+ .setSortDatanodesInPipeline(sortDatanodesInPipeline)
+ .setAcls(acls);
+ }
+
/**
* Builder class of OmKeyArgs.
*/
diff --git a/hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmBucketInfo.java b/hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmBucketInfo.java
index 15468c7..650fc91 100644
--- a/hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmBucketInfo.java
+++ b/hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmBucketInfo.java
@@ -42,10 +42,21 @@ public class TestOmBucketInfo {
.setStorageType(StorageType.ARCHIVE)
.build();
- OmBucketInfo afterSerialization =
- OmBucketInfo.getFromProtobuf(bucket.getProtobuf());
+ Assert.assertEquals(bucket,
+ OmBucketInfo.getFromProtobuf(bucket.getProtobuf()));
+ }
+
+ @Test
+ public void protobufConversionOfBucketLink() {
+ OmBucketInfo bucket = OmBucketInfo.newBuilder()
+ .setBucketName("bucket")
+ .setVolumeName("vol1")
+ .setSourceVolume("otherVol")
+ .setSourceBucket("someBucket")
+ .build();
- Assert.assertEquals(bucket, afterSerialization);
+ Assert.assertEquals(bucket,
+ OmBucketInfo.getFromProtobuf(bucket.getProtobuf()));
}
@Test
@@ -66,7 +77,10 @@ public class TestOmBucketInfo {
/* Clone an omBucketInfo. */
OmBucketInfo cloneBucketInfo = omBucketInfo.copyObject();
- Assert.assertEquals(omBucketInfo, cloneBucketInfo);
+ Assert.assertNotSame(omBucketInfo, cloneBucketInfo);
+ Assert.assertEquals("Expected " + omBucketInfo + " and " + cloneBucketInfo
+ + " to be equal",
+ omBucketInfo, cloneBucketInfo);
/* Reset acl & check not equal. */
omBucketInfo.setAcls(Collections.singletonList(new OzoneAcl(
diff --git a/hadoop-ozone/dist/src/main/compose/ozone/test.sh b/hadoop-ozone/dist/src/main/compose/ozone/test.sh
index e0b1d62..c40339e 100755
--- a/hadoop-ozone/dist/src/main/compose/ozone/test.sh
+++ b/hadoop-ozone/dist/src/main/compose/ozone/test.sh
@@ -26,24 +26,24 @@ source "$COMPOSE_DIR/../testlib.sh"
start_docker_env
-#Due to the limitation of the current auditparser test, it should be the
-#first test in a clean cluster.
-
-#Disabling for now, audit parser tool during parse getting exception.
-#execute_robot_test om auditparser
-
execute_robot_test scm lib
+execute_robot_test scm ozone-lib
execute_robot_test scm basic
execute_robot_test scm gdpr
-execute_robot_test scm -v SCHEME:ofs ozonefs/ozonefs.robot
-execute_robot_test scm -v SCHEME:o3fs ozonefs/ozonefs.robot
+for scheme in ofs o3fs; do
+ for bucket in link bucket; do
+ execute_robot_test scm -v SCHEME:${scheme} -v BUCKET_TYPE:${bucket} ozonefs/ozonefs.robot
+ done
+done
execute_robot_test scm security/ozone-secure-token.robot
-execute_robot_test scm s3
+for bucket in link generated; do
+ execute_robot_test scm -v BUCKET:${bucket} s3
+done
execute_robot_test scm recon
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh b/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
index 9c3f3ab..ce50fa0 100755
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
@@ -31,10 +31,15 @@ execute_robot_test scm basic
execute_robot_test scm security
-execute_robot_test scm -v SCHEME:ofs ozonefs/ozonefs.robot
-execute_robot_test scm -v SCHEME:o3fs ozonefs/ozonefs.robot
-
-execute_robot_test s3g s3
+for scheme in ofs o3fs; do
+ for bucket in link bucket; do
+ execute_robot_test scm -v SCHEME:${scheme} -v BUCKET_TYPE:${bucket} ozonefs/ozonefs.robot
+ done
+done
+
+for bucket in link generated; do
+ execute_robot_test s3g -v BUCKET:${bucket} s3
+done
#expects 4 pipelines, should be run before
#admincli which creates STANDALONE pipeline
diff --git a/hadoop-ozone/dist/src/main/smoketest/basic/links.robot b/hadoop-ozone/dist/src/main/smoketest/basic/links.robot
new file mode 100644
index 0000000..71c046e
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/smoketest/basic/links.robot
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test bucket links via Ozone CLI
+Library OperatingSystem
+Resource ../commonlib.robot
+Resource ../ozone-lib/shell.robot
+Test Setup Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab
+Test Timeout 2 minute
+Suite Setup Create volumes
+
+*** Variables ***
+${prefix} generated
+
+*** Keywords ***
+Create volumes
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${source} ${random}-source
+ Set Suite Variable ${target} ${random}-target
+ Execute ozone sh volume create ${source}
+ Execute ozone sh volume create ${target}
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Setup ACL tests
+
+Setup ACL tests
+ Execute ozone sh bucket create ${source}/readable-bucket
+ Execute ozone sh key put ${source}/readable-bucket/key-in-readable-bucket /etc/passwd
+ Execute ozone sh bucket create ${source}/unreadable-bucket
+ Execute ozone sh bucket link ${source}/readable-bucket ${target}/readable-link
+ Execute ozone sh bucket link ${source}/readable-bucket ${target}/unreadable-link
+ Execute ozone sh bucket link ${source}/unreadable-bucket ${target}/link-to-unreadable-bucket
+ Execute ozone sh volume addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}
+ Execute ozone sh volume addacl --acl user:testuser2/scm@EXAMPLE.COM:rl ${source}
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:rl ${source}/readable-bucket
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}/readable-link
+ Execute ozone sh bucket addacl --acl user:testuser2/scm@EXAMPLE.COM:r ${target}/link-to-unreadable-bucket
+
+Can follow link with read access
+ Execute kdestroy
+ Run Keyword Kinit test user testuser2 testuser2.keytab
+ ${result} = Execute And Ignore Error ozone sh key list ${target}/readable-link
+ Should Contain ${result} key-in-readable-bucket
+
+Cannot follow link without read access
+ Execute kdestroy
+ Run Keyword Kinit test user testuser2 testuser2.keytab
+ ${result} = Execute And Ignore Error ozone sh key list ${target}/unreadable-link
+ Should Contain ${result} PERMISSION_DENIED
+
+ACL verified on source bucket
+ Execute kdestroy
+ Run Keyword Kinit test user testuser2 testuser2.keytab
+ ${result} = Execute ozone sh bucket info ${target}/link-to-unreadable-bucket
+ Should Contain ${result} link-to-unreadable-bucket
+ Should Not Contain ${result} PERMISSION_DENIED
+ ${result} = Execute And Ignore Error ozone sh key list ${target}/link-to-unreadable-bucket
+ Should Contain ${result} PERMISSION_DENIED
+
+*** Test Cases ***
+Link to non-existent bucket
+ Execute ozone sh bucket link ${source}/no-such-bucket ${target}/dangling-link
+ ${result} = Execute And Ignore Error ozone sh key list ${target}/dangling-link
+ Should Contain ${result} BUCKET_NOT_FOUND
+
+Key create passthrough
+ Execute ozone sh bucket link ${source}/bucket1 ${target}/link1
+ Execute ozone sh bucket create ${source}/bucket1
+ Execute ozone sh key put ${target}/link1/key1 /etc/passwd
+ Key Should Match Local File ${target}/link1/key1 /etc/passwd
+
+Key read passthrough
+ Execute ozone sh key put ${source}/bucket1/key2 /opt/hadoop/NOTICE.txt
+ Key Should Match Local File ${source}/bucket1/key2 /opt/hadoop/NOTICE.txt
+
+Key list passthrough
+ ${target_list} = Execute ozone sh key list ${target}/link1 | jq -r '.name'
+ ${source_list} = Execute ozone sh key list ${source}/bucket1 | jq -r '.name'
+ Should Be Equal ${target_list} ${source_list}
+ Should Contain ${source_list} key1
+ Should Contain ${source_list} key2
+
+Key delete passthrough
+ Execute ozone sh key delete ${target}/link1/key2
+ ${source_list} = Execute ozone sh key list ${source}/bucket1 | jq -r '.name'
+ Should Not Contain ${source_list} key2
+
+Bucket list contains links
+ ${result} = Execute ozone sh bucket list ${target}
+ Should Contain ${result} link1
+ Should Contain ${result} dangling-link
+
+Bucket info shows source
+ ${result} = Execute ozone sh bucket info ${target}/link1 | jq -r '.sourceVolume, .sourceBucket' | xargs
+ Should Be Equal ${result} ${source} bucket1
+
+Source and target have separate ACLs
+ Execute ozone sh bucket addacl --acl user:user1:rwxy ${target}/link1
+ Verify ACL bucket ${target}/link1 USER user1 READ WRITE READ_ACL WRITE_ACL
+ Verify ACL bucket ${source}/bucket1 USER user1 ${EMPTY}
+
+ Execute ozone sh bucket addacl --acl group:group2:r ${source}/bucket1
+ Verify ACL bucket ${target}/link1 GROUP group2 ${EMPTY}
+ Verify ACL bucket ${source}/bucket1 GROUP group2 READ
+
+Buckets and links share namespace
+ Execute ozone sh bucket link ${source}/bucket2 ${target}/link2
+ ${result} = Execute And Ignore Error ozone sh bucket create ${target}/link2
+ Should Contain ${result} BUCKET_ALREADY_EXISTS
+
+ Execute ozone sh bucket create ${target}/bucket3
+ ${result} = Execute And Ignore Error ozone sh bucket link ${source}/bucket1 ${target}/bucket3
+ Should Contain ${result} BUCKET_ALREADY_EXISTS
+
+Can follow link with read access
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Can follow link with read access
+
+Cannot follow link without read access
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Cannot follow link without read access
+
+ACL verified on source bucket
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' ACL verified on source bucket
+
+Loop in link chain is detected
+ Execute ozone sh bucket link ${target}/loop1 ${target}/loop2
+ Execute ozone sh bucket link ${target}/loop2 ${target}/loop3
+ Execute ozone sh bucket link ${target}/loop3 ${target}/loop1
+ ${result} = Execute And Ignore Error ozone sh key list ${target}/loop2
+ Should Contain ${result} DETECTED_LOOP
+
+Multiple links to same bucket are allowed
+ Execute ozone sh bucket link ${source}/bucket1 ${target}/link3
+ Execute ozone sh key put ${target}/link3/key3 /etc/group
+ Key Should Match Local File ${target}/link1/key3 /etc/group
+
+Source bucket not affected by deleting link
+ Execute ozone sh bucket delete ${target}/link1
+ ${bucket_list} = Execute ozone sh bucket list ${target}
+ Should Not Contain ${bucket_list} link1
+ ${source_list} = Execute ozone sh key list ${source}/bucket1 | jq -r '.name'
+ Should Contain ${source_list} key1
diff --git a/hadoop-ozone/dist/src/main/smoketest/commonlib.robot b/hadoop-ozone/dist/src/main/smoketest/commonlib.robot
index 407111a..bf3b3e9 100644
--- a/hadoop-ozone/dist/src/main/smoketest/commonlib.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/commonlib.robot
@@ -18,44 +18,14 @@ Library OperatingSystem
Library String
Library BuiltIn
+Resource lib/os.robot
+
*** Variables ***
${SECURITY_ENABLED} false
${OM_HA_PARAM} ${EMPTY}
${OM_SERVICE_ID} om
*** Keywords ***
-Execute
- [arguments] ${command}
- ${rc} ${output} = Run And Return Rc And Output ${command}
- Log ${output}
- Should Be Equal As Integers ${rc} 0
- [return] ${output}
-
-Execute And Ignore Error
- [arguments] ${command}
- ${rc} ${output} = Run And Return Rc And Output ${command}
- Log ${output}
- [return] ${output}
-
-Execute and checkrc
- [arguments] ${command} ${expected_error_code}
- ${rc} ${output} = Run And Return Rc And Output ${command}
- Log ${output}
- Should Be Equal As Integers ${rc} ${expected_error_code}
- [return] ${output}
-
-Compare files
- [arguments] ${file1} ${file2}
- ${checksumbefore} = Execute md5sum ${file1} | awk '{print $1}'
- ${checksumafter} = Execute md5sum ${file2} | awk '{print $1}'
- Should Be Equal ${checksumbefore} ${checksumafter}
-
-Install aws cli
- ${rc} ${output} = Run And Return Rc And Output which apt-get
- Run Keyword if '${rc}' == '0' Install aws cli s3 debian
- ${rc} ${output} = Run And Return Rc And Output yum --help
- Run Keyword if '${rc}' == '0' Install aws cli s3 centos
-
Kinit HTTP user
${hostname} = Execute hostname
Wait Until Keyword Succeeds 2min 10sec Execute kinit -k HTTP/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab
diff --git a/hadoop-ozone/dist/src/main/smoketest/commonlib.robot b/hadoop-ozone/dist/src/main/smoketest/lib/os.robot
similarity index 57%
copy from hadoop-ozone/dist/src/main/smoketest/commonlib.robot
copy to hadoop-ozone/dist/src/main/smoketest/lib/os.robot
index 407111a..af927f9 100644
--- a/hadoop-ozone/dist/src/main/smoketest/commonlib.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/lib/os.robot
@@ -14,22 +14,12 @@
# limitations under the License.
*** Settings ***
-Library OperatingSystem
-Library String
-Library BuiltIn
-
-*** Variables ***
-${SECURITY_ENABLED} false
-${OM_HA_PARAM} ${EMPTY}
-${OM_SERVICE_ID} om
+Library OperatingSystem
*** Keywords ***
Execute
[arguments] ${command}
- ${rc} ${output} = Run And Return Rc And Output ${command}
- Log ${output}
- Should Be Equal As Integers ${rc} 0
- [return] ${output}
+ Run Keyword And Return Execute and checkrc ${command} 0
Execute And Ignore Error
[arguments] ${command}
@@ -50,18 +40,10 @@ Compare files
${checksumafter} = Execute md5sum ${file2} | awk '{print $1}'
Should Be Equal ${checksumbefore} ${checksumafter}
-Install aws cli
- ${rc} ${output} = Run And Return Rc And Output which apt-get
- Run Keyword if '${rc}' == '0' Install aws cli s3 debian
- ${rc} ${output} = Run And Return Rc And Output yum --help
- Run Keyword if '${rc}' == '0' Install aws cli s3 centos
-
-Kinit HTTP user
- ${hostname} = Execute hostname
- Wait Until Keyword Succeeds 2min 10sec Execute kinit -k HTTP/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab
-
-Kinit test user
- [arguments] ${user} ${keytab}
- ${hostname} = Execute hostname
- Set Suite Variable ${TEST_USER} ${user}/${hostname}@EXAMPLE.COM
- Wait Until Keyword Succeeds 2min 10sec Execute kinit -k ${user}/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/${keytab}
+Create Random File
+ ${postfix} = Generate Random String 5 [NUMBERS]
+ ${tmpfile} = Set Variable /tmp/tempfile-${postfix}
+ File Should Not Exist ${tmpfile}
+ ${content} = Set Variable "Random string"
+ Create File ${tmpfile} ${content}
+ [Return] ${tmpfile}
diff --git a/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot b/hadoop-ozone/dist/src/main/smoketest/lib/os_tests.robot
similarity index 53%
copy from hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot
copy to hadoop-ozone/dist/src/main/smoketest/lib/os_tests.robot
index bcba30d..dd4beaf 100644
--- a/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/lib/os_tests.robot
@@ -14,24 +14,25 @@
# limitations under the License.
*** Settings ***
-Documentation S3 gateway test with aws cli
-Library OperatingSystem
-Library String
-Resource ../commonlib.robot
-Resource commonawslib.robot
-Test Timeout 5 minutes
-Suite Setup Setup s3 tests
-
-*** Variables ***
-${ENDPOINT_URL} http://s3g:9878
-${BUCKET} generated
+Resource os.robot
+
*** Test Cases ***
-Delete existing bucket
-# Bucket already is created in Test Setup.
- Execute AWSS3APICli delete-bucket --bucket ${BUCKET}
+Execute
+ ${output} = Execute echo 42
+ Should Be Equal ${output} 42
+
+Execute failing command
+ Run Keyword And Expect Error * Execute false
+
+Execute And Ignore Error
+ ${output} = Execute And Ignore Error echo 123 && false
+ Should Be Equal ${output} 123
+
+Execute and checkrc
+ ${output} = Execute and checkrc echo failure && exit 1 1
+ Should Be Equal ${output} failure
-Delete non-existent bucket
- ${result} = Execute AWSS3APICli and checkrc delete-bucket --bucket nosuchbucket 255
- Should contain ${result} NoSuchBucket
+Execute and checkrc RC mismatch
+ Run Keyword And Expect Error * Execute and checkrc echo failure && exit 3 1
diff --git a/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
new file mode 100644
index 0000000..2e56ae4
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Resource ../lib/os.robot
+Library String
+
+
+*** Keywords ***
+Bucket Exists
+ [arguments] ${bucket}
+ ${rc} ${output} = Run And Return Rc And Output timeout 15 ozone sh bucket info ${bucket}
+ Return From Keyword If ${rc} != 0 ${FALSE}
+ Return From Keyword If 'VOLUME_NOT_FOUND' in '''${output}''' ${FALSE}
+ Return From Keyword If 'BUCKET_NOT_FOUND' in '''${output}''' ${FALSE}
+ [Return] ${TRUE}
+
+Compare Key With Local File
+ [arguments] ${key} ${file}
+ ${postfix} = Generate Random String 5 [NUMBERS]
+ ${tmpfile} = Set Variable /tmp/tempkey-${postfix}
+ Execute ozone sh key get -f ${key} ${tmpfile}
+ ${rc} = Run And Return Rc diff -q ${file} ${tmpfile}
+ Execute rm -f ${tmpfile}
+ ${result} = Set Variable If ${rc} == 0 ${TRUE} ${FALSE}
+ [Return] ${result}
+
+Key Should Match Local File
+ [arguments] ${key} ${file}
+ ${matches} = Compare Key With Local File ${key} ${file}
+ Should Be True ${matches}
+
+Verify ACL
+ [arguments] ${object_type} ${object} ${type} ${name} ${acls}
+ ${actual_acls} = Execute ozone sh ${object_type} getacl ${object} | jq -r '.[] | select(.type == "${type}") | select(.name == "${name}") | .aclList[]' | xargs
+ Should Be Equal ${acls} ${actual_acls}
diff --git a/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell_tests.robot b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell_tests.robot
new file mode 100644
index 0000000..56fbcf8
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell_tests.robot
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Resource ../lib/os.robot
+Resource shell.robot
+
+
+*** Variables ***
+${OM_SERVICE_ID} om
+
+
+*** Test Cases ***
+
+Bucket Exists should not if No Such Volume
+ ${exists} = Bucket Exists o3://${OM_SERVICE_ID}/no-such-volume/any-bucket
+ Should Be Equal ${exists} ${FALSE}
+
+Bucket Exists should not if No Such Bucket
+ Execute And Ignore Error ozone sh volume create o3://${OM_SERVICE_ID}/vol1
+ ${exists} = Bucket Exists o3://${OM_SERVICE_ID}/vol1/no-such-bucket
+ Should Be Equal ${exists} ${FALSE}
+
+Bucket Exists
+ Execute And Ignore Error ozone sh bucket create o3://${OM_SERVICE_ID}/vol1/bucket
+ ${exists} = Bucket Exists o3://${OM_SERVICE_ID}/vol1/bucket
+ Should Be Equal ${exists} ${TRUE}
+
+Bucket Exists should not if No Such OM service
+ ${exists} = Bucket Exists o3://no-such-host/any-volume/any-bucket
+ Should Be Equal ${exists} ${FALSE}
+
+
+Key Should Match Local File
+ [Setup] Execute ozone sh key put o3://${OM_SERVICE_ID}/vol1/bucket/passwd /etc/passwd
+ Key Should Match Local File o3://${OM_SERVICE_ID}/vol1/bucket/passwd /etc/passwd
+
+Compare Key With Local File with Different File
+ ${random_file} = Create Random File
+ ${matches} = Compare Key With Local File o3://${OM_SERVICE_ID}/vol1/bucket/passwd ${random_file}
+ Should Be Equal ${matches} ${FALSE}
+ [Teardown] Remove File ${random_file}
+
+Compare Key With Local File if File Does Not Exist
+ ${matches} = Compare Key With Local File o3://${OM_SERVICE_ID}/vol1/bucket/passwd /no-such-file
+ Should Be Equal ${matches} ${FALSE}
diff --git a/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot b/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
index 6d0042b..450f1b6 100644
--- a/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
@@ -19,7 +19,7 @@ Library OperatingSystem
Resource ../commonlib.robot
Resource setup.robot
Test Timeout 5 minutes
-Suite Setup Setup ${BUCKET_TYPE}s for FS test
+Suite Setup Setup for FS test
*** Test Cases ***
List root
diff --git a/hadoop-ozone/dist/src/main/smoketest/ozonefs/setup.robot b/hadoop-ozone/dist/src/main/smoketest/ozonefs/setup.robot
index 16e059e..441822d 100644
--- a/hadoop-ozone/dist/src/main/smoketest/ozonefs/setup.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/ozonefs/setup.robot
@@ -29,12 +29,12 @@ ${BUCKET_IN_VOL2} ${BUCKET_TYPE}3-${SCHEME}
${DEEP_DIR} test/${SCHEME}/dir
*** Keywords ***
-Setup buckets for FS test
+Setup for FS test
Create volumes for FS test
- Create buckets for FS test
+ Run Keyword Create ${BUCKET_TYPE}s for FS test
Sanity check for FS test
Assign suite vars for FS test
- Log Completed setup for ${SCHEME} tests in ${VOLUME}/${BUCKET} using FS base URL: ${BASE_URL}
+ Log Completed setup for ${SCHEME} tests with ${BUCKET_TYPE}s in ${VOLUME}/${BUCKET} using FS base URL: ${BASE_URL}
Create volumes for FS test
Execute And Ignore Error ozone sh volume create ${VOLUME} --quota 100TB
@@ -45,6 +45,16 @@ Create buckets for FS test
Execute ozone sh bucket create ${VOLUME}/${BUCKET2}
Execute ozone sh bucket create ${VOL2}/${BUCKET_IN_VOL2}
+Create links for FS test
+ Execute And Ignore Error ozone sh volume create ${VOLUME}-src --quota 100TB
+ Execute And Ignore Error ozone sh volume create ${VOL2}-src --quota 100TB
+ Execute ozone sh bucket create ${VOLUME}-src/${BUCKET}-src
+ Execute ozone sh bucket create ${VOLUME}-src/${BUCKET2}-src
+ Execute ozone sh bucket create ${VOL2}-src/${BUCKET_IN_VOL2}-src
+ Execute ozone sh bucket link ${VOLUME}-src/${BUCKET}-src ${VOLUME}/${BUCKET}
+ Execute ozone sh bucket link ${VOLUME}-src/${BUCKET2}-src ${VOLUME}/${BUCKET2}
+ Execute ozone sh bucket link ${VOL2}-src/${BUCKET_IN_VOL2}-src ${VOL2}/${BUCKET_IN_VOL2}
+
Sanity check for FS test
${result} = Execute ozone sh volume list
Should contain ${result} ${VOLUME}
diff --git a/hadoop-ozone/dist/src/main/smoketest/robot.robot b/hadoop-ozone/dist/src/main/smoketest/robot.robot
new file mode 100644
index 0000000..d677ef3
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/smoketest/robot.robot
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Smoketest for Robot functions
+Resource commonlib.robot
+Test Timeout 5 minutes
+
+*** Test Cases ***
+
+Ensure Leading without Leading
+ ${result} = Ensure Leading / a/b
+ Should Be Equal ${result} /a/b
+
+Ensure Leading with Leading
+ ${result} = Ensure Leading _ _a_b_c
+ Should Be Equal ${result} _a_b_c
+
+Ensure Leading for empty
+ ${result} = Ensure Leading | ${EMPTY}
+ Should Be Equal ${result} |
+
+
+Ensure Trailing without Trailing
+ ${result} = Ensure Trailing . x.y.z
+ Should Be Equal ${result} x.y.z.
+
+Ensure Trailing with Trailing
+ ${result} = Ensure Trailing x axbxcx
+ Should Be Equal ${result} axbxcx
+
+Ensure Trailing for empty
+ ${result} = Ensure Trailing = ${EMPTY}
+ Should Be Equal ${result} =
+
+
+Format o3fs URL without path
+ ${result} = Format o3fs URL vol1 bucket1
+ Should Be Equal ${result} o3fs://bucket1.vol1/
+
+Format o3fs URL with path
+ ${result} = Format o3fs URL vol1 bucket1 dir/file
+ Should Be Equal ${result} o3fs://bucket1.vol1/dir/file
+
+
+Format ofs URL without path
+ ${result} = Format ofs URL vol1 bucket1
+ Should Be Equal ${result} ofs://vol1/bucket1
+
+Format ofs URL with path
+ ${result} = Format ofs URL vol1 bucket1 dir/file
+ Should Be Equal ${result} ofs://vol1/bucket1/dir/file
+
+
+Format FS URL with ofs scheme
+ ${result} = Format FS URL ofs vol1 bucket1
+ ${expected} = Format ofs URL vol1 bucket1
+ Should Be Equal ${result} ${expected}
+
+Format FS URL with o3fs scheme
+ ${result} = Format FS URL o3fs vol1 bucket1
+ ${expected} = Format o3fs URL vol1 bucket1
+ Should Be Equal ${result} ${expected}
+
+Format FS URL with unsupported scheme
+ ${result} = Run Keyword And Expect Error * Format FS URL http org apache
+ Should Contain ${result} http
+ Should Contain ${result} nsupported
+
diff --git a/hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot b/hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
index 004a496..1c6827a 100644
--- a/hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
@@ -88,7 +88,7 @@ Test Multipart Upload Complete
#read file and check the key
${result} = Execute AWSS3ApiCli get-object --bucket ${BUCKET} --key multipartKey1 /tmp/multipartKey1.result
- Execute cat /tmp/part1 /tmp/part2 >> /tmp/multipartKey1
+ Execute cat /tmp/part1 /tmp/part2 > /tmp/multipartKey1
Compare files /tmp/multipartKey1 /tmp/multipartKey1.result
Test Multipart Upload Complete Entity too small
@@ -156,7 +156,7 @@ Test Multipart Upload Complete Invalid part errors and complete mpu with few par
Should contain ${result} ETag
${result} = Execute AWSS3ApiCli get-object --bucket ${BUCKET} --key multipartKey3 /tmp/multipartKey3.result
- Execute cat /tmp/part1 /tmp/part3 >> /tmp/multipartKey3
+ Execute cat /tmp/part1 /tmp/part3 > /tmp/multipartKey3
Compare files /tmp/multipartKey3 /tmp/multipartKey3.result
Test abort Multipart upload
@@ -237,7 +237,6 @@ Test Multipart Upload Put With Copy
Should contain ${result} UploadId
${result} = Execute AWSS3APICli upload-part-copy --bucket ${BUCKET} --key copytest/destination --upload-id ${uploadID} --part-number 1 --copy-source ${BUCKET}/copytest/source
- Should contain ${result} ${BUCKET}
Should contain ${result} ETag
Should contain ${result} LastModified
${eTag1} = Execute and checkrc echo '${result}' | jq -r '.CopyPartResult.ETag' 0
@@ -260,13 +259,11 @@ Test Multipart Upload Put With Copy and range
Should contain ${result} UploadId
${result} = Execute AWSS3APICli upload-part-copy --bucket ${BUCKET} --key copyrange/destination --upload-id ${uploadID} --part-number 1 --copy-source ${BUCKET}/copyrange/source --copy-source-range bytes=0-10485758
- Should contain ${result} ${BUCKET}
Should contain ${result} ETag
Should contain ${result} LastModified
${eTag1} = Execute and checkrc echo '${result}' | jq -r '.CopyPartResult.ETag' 0
${result} = Execute AWSS3APICli upload-part-copy --bucket ${BUCKET} --key copyrange/destination --upload-id ${uploadID} --part-number 2 --copy-source ${BUCKET}/copyrange/source --copy-source-range bytes=10485758-10485760
- Should contain ${result} ${BUCKET}
Should contain ${result} ETag
Should contain ${result} LastModified
${eTag2} = Execute and checkrc echo '${result}' | jq -r '.CopyPartResult.ETag' 0
diff --git a/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot b/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot
index bcba30d..ce7b825 100644
--- a/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/s3/bucketdelete.robot
@@ -23,14 +23,20 @@ Test Timeout 5 minutes
Suite Setup Setup s3 tests
*** Variables ***
-${ENDPOINT_URL} http://s3g:9878
${BUCKET} generated
+${ENDPOINT_URL} http://s3g:9878
+
+*** Keywords ***
+Create bucket to be deleted
+ ${bucket} = Run Keyword if '${BUCKET}' == 'link' Create link to-be-deleted
+ ... ELSE Run Keyword Create bucket
+ [return] ${bucket}
*** Test Cases ***
Delete existing bucket
-# Bucket already is created in Test Setup.
- Execute AWSS3APICli delete-bucket --bucket ${BUCKET}
+ ${bucket} = Create bucket to be deleted
+ Execute AWSS3APICli delete-bucket --bucket ${bucket}
Delete non-existent bucket
${result} = Execute AWSS3APICli and checkrc delete-bucket --bucket nosuchbucket 255
diff --git a/hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot b/hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
index 4595587..c263988 100644
--- a/hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
@@ -15,12 +15,13 @@
*** Settings ***
Resource ../commonlib.robot
-Resource ../commonlib.robot
+Resource ../ozone-lib/shell.robot
*** Variables ***
+${ENDPOINT_URL} http://s3g:9878
${OZONE_S3_HEADER_VERSION} v4
${OZONE_S3_SET_CREDENTIALS} true
-${BUCKET} bucket-999
+${BUCKET} generated
*** Keywords ***
Execute AWSS3APICli
@@ -38,6 +39,12 @@ Execute AWSS3Cli
${output} = Execute aws s3 --endpoint-url ${ENDPOINT_URL} ${command}
[return] ${output}
+Install aws cli
+ ${rc} ${output} = Run And Return Rc And Output which apt-get
+ Run Keyword if '${rc}' == '0' Install aws cli s3 debian
+ ${rc} ${output} = Run And Return Rc And Output yum --help
+ Run Keyword if '${rc}' == '0' Install aws cli s3 centos
+
Install aws cli s3 centos
Execute sudo -E yum install -y awscli
@@ -73,8 +80,9 @@ Setup dummy credentials for S3
Create bucket
${postfix} = Generate Random String 5 [NUMBERS]
- Set Suite Variable ${BUCKET} bucket-${postfix}
- Create bucket with name ${BUCKET}
+ ${bucket} = Set Variable bucket-${postfix}
+ Create bucket with name ${bucket}
+ [Return] ${bucket}
Create bucket with name
[Arguments] ${bucket}
@@ -87,4 +95,19 @@ Setup s3 tests
Run Keyword if '${OZONE_S3_SET_CREDENTIALS}' == 'true' Setup v4 headers
${result} = Execute And Ignore Error ozone sh volume create o3://${OM_SERVICE_ID}/s3v
Should not contain ${result} Failed
- Run Keyword if '${BUCKET}' == 'generated' Create bucket
+ ${BUCKET} = Run Keyword if '${BUCKET}' == 'generated' Create bucket
+ ... ELSE Set Variable ${BUCKET}
+ Set Suite Variable ${BUCKET}
+ Run Keyword if '${BUCKET}' == 'link' Setup links for S3 tests
+
+Setup links for S3 tests
+ ${exists} = Bucket Exists o3://${OM_SERVICE_ID}/s3v/link
+ Return From Keyword If ${exists}
+ Execute ozone sh volume create o3://${OM_SERVICE_ID}/legacy
+ Execute ozone sh bucket create o3://${OM_SERVICE_ID}/legacy/source-bucket
+ Create link link
+
+Create link
+ [arguments] ${bucket}
+ Execute ozone sh bucket link o3://${OM_SERVICE_ID}/legacy/source-bucket o3://${OM_SERVICE_ID}/s3v/${bucket}
+ [return] ${bucket}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmMetrics.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmMetrics.java
index d4594ef..b80e357 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmMetrics.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmMetrics.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with this
* work for additional information regarding copyright ownership. The ASF
@@ -19,11 +19,11 @@ package org.apache.hadoop.ozone.om;
import static org.apache.hadoop.test.MetricsAsserts.assertCounter;
import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyInt;
import static org.mockito.Matchers.anyLong;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
+import java.util.Collections;
import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hdds.client.BlockID;
@@ -32,6 +32,7 @@ import org.apache.hadoop.hdds.scm.HddsWhiteboxTestUtils;
import org.apache.hadoop.metrics2.MetricsRecordBuilder;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
import org.apache.hadoop.ozone.om.helpers.OmKeyArgs;
import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
import org.junit.After;
@@ -44,7 +45,6 @@ import org.mockito.Mockito;
/**
* Test for OM metrics.
*/
-@SuppressWarnings("deprecation")
public class TestOmMetrics {
/**
@@ -62,8 +62,6 @@ public class TestOmMetrics {
/**
* Create a MiniDFSCluster for testing.
- *
- * @throws IOException
*/
@Before
public void setup() throws Exception {
@@ -233,20 +231,28 @@ public class TestOmMetrics {
KeyManager keyManager = (KeyManager) HddsWhiteboxTestUtils
.getInternalState(ozoneManager, "keyManager");
KeyManager mockKm = Mockito.spy(keyManager);
-
- Mockito.doReturn(null).when(mockKm).openKey(null);
- Mockito.doNothing().when(mockKm).deleteKey(null);
- Mockito.doReturn(null).when(mockKm).lookupKey(null, "");
- Mockito.doReturn(null).when(mockKm).listKeys(null, null, null, null, 0);
- Mockito.doReturn(null).when(mockKm).listTrash(
- null, null, null, null, 0);
- Mockito.doNothing().when(mockKm).commitKey(any(OmKeyArgs.class), anyLong());
- Mockito.doReturn(null).when(mockKm).initiateMultipartUpload(
- any(OmKeyArgs.class));
+ BucketManager mockBm = Mockito.mock(BucketManager.class);
+
+ OmBucketInfo mockBucket = OmBucketInfo.newBuilder()
+ .setVolumeName("").setBucketName("")
+ .build();
+ Mockito.when(mockBm.getBucketInfo(any(), any())).thenReturn(mockBucket);
+ Mockito.doReturn(null).when(mockKm).openKey(any());
+ Mockito.doNothing().when(mockKm).deleteKey(any());
+ Mockito.doReturn(null).when(mockKm).lookupKey(any(), any());
+ Mockito.doReturn(null).when(mockKm).listKeys(any(), any(), any(), any(),
+ anyInt());
+ Mockito.doReturn(null).when(mockKm).listTrash(any(), any(), any(), any(),
+ anyInt());
+ Mockito.doNothing().when(mockKm).commitKey(any(), anyLong());
+ Mockito.doReturn(null).when(mockKm).initiateMultipartUpload(any());
HddsWhiteboxTestUtils.setInternalState(
+ ozoneManager, "bucketManager", mockBm);
+ HddsWhiteboxTestUtils.setInternalState(
ozoneManager, "keyManager", mockKm);
- doKeyOps();
+ OmKeyArgs keyArgs = createKeyArgs();
+ doKeyOps(keyArgs);
MetricsRecordBuilder omMetrics = getMetrics("OMMetrics");
assertCounter("NumKeyOps", 7L, omMetrics);
@@ -259,34 +265,32 @@ public class TestOmMetrics {
assertCounter("NumInitiateMultipartUploads", 1L, omMetrics);
- ozoneManager.openKey(null);
- ozoneManager.commitKey(createKeyArgs(), 0);
- ozoneManager.openKey(null);
- ozoneManager.commitKey(createKeyArgs(), 0);
- ozoneManager.openKey(null);
- ozoneManager.commitKey(createKeyArgs(), 0);
- ozoneManager.deleteKey(null);
+ ozoneManager.openKey(keyArgs);
+ ozoneManager.commitKey(keyArgs, 0);
+ ozoneManager.openKey(keyArgs);
+ ozoneManager.commitKey(keyArgs, 0);
+ ozoneManager.openKey(keyArgs);
+ ozoneManager.commitKey(keyArgs, 0);
+ ozoneManager.deleteKey(keyArgs);
omMetrics = getMetrics("OMMetrics");
assertCounter("NumKeys", 2L, omMetrics);
// inject exception to test for Failure Metrics
- Mockito.doThrow(exception).when(mockKm).openKey(null);
- Mockito.doThrow(exception).when(mockKm).deleteKey(null);
- Mockito.doThrow(exception).when(mockKm).lookupKey(null, "");
+ Mockito.doThrow(exception).when(mockKm).openKey(any());
+ Mockito.doThrow(exception).when(mockKm).deleteKey(any());
+ Mockito.doThrow(exception).when(mockKm).lookupKey(any(), any());
Mockito.doThrow(exception).when(mockKm).listKeys(
- null, null, null, null, 0);
+ any(), any(), any(), any(), anyInt());
Mockito.doThrow(exception).when(mockKm).listTrash(
- null, null, null, null, 0);
- Mockito.doThrow(exception).when(mockKm).commitKey(any(OmKeyArgs.class),
- anyLong());
- Mockito.doThrow(exception).when(mockKm).initiateMultipartUpload(
- any(OmKeyArgs.class));
+ any(), any(), any(), any(), anyInt());
+ Mockito.doThrow(exception).when(mockKm).commitKey(any(), anyLong());
+ Mockito.doThrow(exception).when(mockKm).initiateMultipartUpload(any());
HddsWhiteboxTestUtils.setInternalState(
ozoneManager, "keyManager", mockKm);
- doKeyOps();
+ doKeyOps(keyArgs);
omMetrics = getMetrics("OMMetrics");
assertCounter("NumKeyOps", 21L, omMetrics);
@@ -380,39 +384,39 @@ public class TestOmMetrics {
/**
* Test key operations with ignoring thrown exception.
*/
- private void doKeyOps() {
+ private void doKeyOps(OmKeyArgs keyArgs) {
try {
- ozoneManager.openKey(null);
+ ozoneManager.openKey(keyArgs);
} catch (IOException ignored) {
}
try {
- ozoneManager.deleteKey(null);
+ ozoneManager.deleteKey(keyArgs);
} catch (IOException ignored) {
}
try {
- ozoneManager.lookupKey(null);
+ ozoneManager.lookupKey(keyArgs);
} catch (IOException ignored) {
}
try {
- ozoneManager.listKeys(null, null, null, null, 0);
+ ozoneManager.listKeys("", "", null, null, 0);
} catch (IOException ignored) {
}
try {
- ozoneManager.listTrash(null, null, null, null, 0);
+ ozoneManager.listTrash("", "", null, null, 0);
} catch (IOException ignored) {
}
try {
- ozoneManager.commitKey(createKeyArgs(), 0);
+ ozoneManager.commitKey(keyArgs, 0);
} catch (IOException ignored) {
}
try {
- ozoneManager.initiateMultipartUpload(null);
+ ozoneManager.initiateMultipartUpload(keyArgs);
} catch (IOException ignored) {
}
@@ -420,12 +424,12 @@ public class TestOmMetrics {
private OmKeyArgs createKeyArgs() {
OmKeyLocationInfo keyLocationInfo = new OmKeyLocationInfo.Builder()
- .setBlockID(new BlockID(new ContainerBlockID(1, 1))).build();
+ .setBlockID(new BlockID(new ContainerBlockID(1, 1)))
+ .build();
keyLocationInfo.setCreateVersion(0);
- List<OmKeyLocationInfo> omKeyLocationInfoList = new ArrayList<>();
- omKeyLocationInfoList.add(keyLocationInfo);
- OmKeyArgs keyArgs = new OmKeyArgs.Builder().setLocationInfoList(
- omKeyLocationInfoList).build();
- return keyArgs;
+
+ return new OmKeyArgs.Builder()
+ .setLocationInfoList(Collections.singletonList(keyLocationInfo))
+ .build();
}
}
diff --git a/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto b/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
index f6eaf38..6859817 100644
--- a/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
+++ b/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
@@ -304,6 +304,8 @@ enum Status {
INVALID_VOLUME_NAME = 61;
PARTIAL_DELETE = 62;
+
+ DETECTED_LOOP_IN_BUCKET_LINKS = 63;
}
/**
@@ -483,6 +485,8 @@ message BucketInfo {
optional uint64 objectID = 9;
optional uint64 updateID = 10;
optional uint64 modificationTime = 11;
+ optional string sourceVolume = 12;
+ optional string sourceBucket = 13;
}
enum StorageTypeProto {
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
index 2e0c6cf..4349d7c 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketManagerImpl.java
@@ -17,7 +17,6 @@
package org.apache.hadoop.ozone.om;
import java.io.IOException;
-import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
@@ -25,7 +24,7 @@ import org.apache.hadoop.crypto.CipherSuite;
import org.apache.hadoop.crypto.CryptoProtocolVersion;
import org.apache.hadoop.crypto.key.KeyProvider;
import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
-import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
import org.apache.hadoop.hdds.protocol.StorageType;
import org.apache.hadoop.ozone.OzoneAcl;
import org.apache.hadoop.ozone.om.exceptions.OMException;
@@ -34,6 +33,7 @@ import org.apache.hadoop.ozone.om.helpers.OmBucketArgs;
import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
import org.apache.hadoop.ozone.om.helpers.OzoneAclUtil;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import org.apache.hadoop.ozone.security.acl.OzoneObj;
import org.apache.hadoop.ozone.security.acl.RequestContext;
import org.apache.hadoop.util.StringUtils;
@@ -41,6 +41,7 @@ import org.apache.hadoop.util.Time;
import com.google.common.base.Preconditions;
import org.iq80.leveldb.DBException;
+import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -136,46 +137,34 @@ public class BucketManagerImpl implements BucketManager {
throw new OMException("Bucket already exist",
OMException.ResultCodes.BUCKET_ALREADY_EXISTS);
}
+
BucketEncryptionKeyInfo bek = bucketInfo.getEncryptionKeyInfo();
- BucketEncryptionKeyInfo.Builder bekb = null;
- if (bek != null) {
- if (kmsProvider == null) {
- throw new OMException("Invalid KMS provider, check configuration " +
- CommonConfigurationKeys.HADOOP_SECURITY_KEY_PROVIDER_PATH,
- OMException.ResultCodes.INVALID_KMS_PROVIDER);
- }
- if (bek.getKeyName() == null) {
- throw new OMException("Bucket encryption key needed.", OMException
- .ResultCodes.BUCKET_ENCRYPTION_KEY_NOT_FOUND);
- }
- // Talk to KMS to retrieve the bucket encryption key info.
- KeyProvider.Metadata metadata = getKMSProvider().getMetadata(
- bek.getKeyName());
- if (metadata == null) {
- throw new OMException("Bucket encryption key " + bek.getKeyName()
- + " doesn't exist.",
- OMException.ResultCodes.BUCKET_ENCRYPTION_KEY_NOT_FOUND);
- }
- // If the provider supports pool for EDEKs, this will fill in the pool
- kmsProvider.warmUpEncryptedKeys(bek.getKeyName());
- bekb = new BucketEncryptionKeyInfo.Builder()
- .setKeyName(bek.getKeyName())
- .setVersion(CryptoProtocolVersion.ENCRYPTION_ZONES)
- .setSuite(CipherSuite.convert(metadata.getCipher()));
- }
- List<OzoneAcl> acls = new ArrayList<>();
- acls.addAll(bucketInfo.getAcls());
- volumeArgs.getAclMap().getDefaultAclList().forEach(
- a -> acls.add(OzoneAcl.fromProtobufWithAccessType(a)));
-
- OmBucketInfo.Builder omBucketInfoBuilder = OmBucketInfo.newBuilder()
- .setVolumeName(bucketInfo.getVolumeName())
- .setBucketName(bucketInfo.getBucketName())
- .setAcls(acls)
- .setStorageType(bucketInfo.getStorageType())
- .setIsVersionEnabled(bucketInfo.getIsVersionEnabled())
- .setCreationTime(Time.now())
- .addAllMetadata(bucketInfo.getMetadata());
+
+ boolean hasSourceVolume = bucketInfo.getSourceVolume() != null;
+ boolean hasSourceBucket = bucketInfo.getSourceBucket() != null;
+
+ if (hasSourceBucket != hasSourceVolume) {
+ throw new OMException("Both source volume and source bucket are " +
+ "required for bucket links",
+ OMException.ResultCodes.INVALID_REQUEST);
+ }
+
+ if (bek != null && hasSourceBucket) {
+ throw new OMException("Encryption cannot be set for bucket links",
+ OMException.ResultCodes.INVALID_REQUEST);
+ }
+
+ BucketEncryptionKeyInfo.Builder bekb =
+ createBucketEncryptionKeyInfoBuilder(bek);
+
+ OmBucketInfo.Builder omBucketInfoBuilder = bucketInfo.toBuilder()
+ .setCreationTime(Time.now());
+
+ List<OzoneManagerProtocolProtos.OzoneAclInfo> defaultAclList =
+ volumeArgs.getAclMap().getDefaultAclList();
+ for (OzoneManagerProtocolProtos.OzoneAclInfo a : defaultAclList) {
+ omBucketInfoBuilder.addAcl(OzoneAcl.fromProtobufWithAccessType(a));
+ }
if (bekb != null) {
omBucketInfoBuilder.setBucketEncryptionKey(bekb.build());
@@ -183,7 +172,14 @@ public class BucketManagerImpl implements BucketManager {
OmBucketInfo omBucketInfo = omBucketInfoBuilder.build();
commitBucketInfoToDB(omBucketInfo);
- LOG.debug("created bucket: {} in volume: {}", bucketName, volumeName);
+ if (hasSourceBucket) {
+ LOG.debug("created link {}/{} to bucket: {}/{}",
+ volumeName, bucketName,
+ omBucketInfo.getSourceVolume(), omBucketInfo.getSourceBucket());
+ } else {
+ LOG.debug("created bucket: {} in volume: {}", bucketName,
+ volumeName);
+ }
} catch (IOException | DBException ex) {
if (!(ex instanceof OMException)) {
LOG.error("Bucket creation failed for bucket:{} in volume:{}",
@@ -199,6 +195,38 @@ public class BucketManagerImpl implements BucketManager {
}
}
+ @Nullable
+ public BucketEncryptionKeyInfo.Builder createBucketEncryptionKeyInfoBuilder(
+ BucketEncryptionKeyInfo bek) throws IOException {
+ BucketEncryptionKeyInfo.Builder bekb = null;
+ if (bek != null) {
+ if (kmsProvider == null) {
+ throw new OMException("Invalid KMS provider, check configuration " +
+ CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH,
+ OMException.ResultCodes.INVALID_KMS_PROVIDER);
+ }
+ if (bek.getKeyName() == null) {
+ throw new OMException("Bucket encryption key needed.", OMException
+ .ResultCodes.BUCKET_ENCRYPTION_KEY_NOT_FOUND);
+ }
+ // Talk to KMS to retrieve the bucket encryption key info.
+ KeyProvider.Metadata metadata = getKMSProvider().getMetadata(
+ bek.getKeyName());
+ if (metadata == null) {
+ throw new OMException("Bucket encryption key " + bek.getKeyName()
+ + " doesn't exist.",
+ OMException.ResultCodes.BUCKET_ENCRYPTION_KEY_NOT_FOUND);
+ }
+ // If the provider supports pool for EDEKs, this will fill in the pool
+ kmsProvider.warmUpEncryptedKeys(bek.getKeyName());
+ bekb = new BucketEncryptionKeyInfo.Builder()
+ .setKeyName(bek.getKeyName())
+ .setVersion(CryptoProtocolVersion.ENCRYPTION_ZONES)
+ .setSuite(CipherSuite.convert(metadata.getCipher()));
+ }
+ return bekb;
+ }
+
private void commitBucketInfoToDB(OmBucketInfo omBucketInfo)
throws IOException {
String dbBucketKey =
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 8c0fcbd..0905d81 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@ -36,15 +36,18 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.conf.StorageUnit;
import org.apache.hadoop.crypto.key.KeyProvider;
import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
@@ -133,6 +136,7 @@ import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils;
import org.apache.hadoop.ozone.om.snapshot.OzoneManagerSnapshotProvider;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.DBUpdatesRequest;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OzoneAclInfo;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ServicePort;
@@ -204,6 +208,7 @@ import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_USER_MAX_VOLUME;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_USER_MAX_VOLUME_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_VOLUME_LISTALL_ALLOWED;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_VOLUME_LISTALL_ALLOWED_DEFAULT;
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.DETECTED_LOOP_IN_BUCKET_LINKS;
import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_AUTH_METHOD;
import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.KEY_NOT_FOUND;
@@ -231,6 +236,7 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
AuditLoggerType.OMLOGGER);
private static final String OM_DAEMON = "om";
+
private static boolean securityEnabled = false;
private OzoneDelegationTokenSecretManager delegationTokenMgr;
private OzoneBlockTokenSecretManager blockTokenMgr;
@@ -2025,22 +2031,29 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
*/
@Override
public OpenKeySession openKey(OmKeyArgs args) throws IOException {
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
try {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} catch (OMException ex) {
// For new keys key checkAccess call will fail as key doesn't exist.
// Check user access for bucket.
if (ex.getResult().equals(KEY_NOT_FOUND)) {
checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} else {
throw ex;
}
}
}
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumKeyAllocates();
return keyManager.openKey(args);
@@ -2048,12 +2061,12 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumKeyAllocateFails();
auditSuccess = false;
AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction.ALLOCATE_KEY,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
- OMAction.ALLOCATE_KEY, (args == null) ? null : args.toAuditMap()));
+ OMAction.ALLOCATE_KEY, auditMap));
}
}
}
@@ -2061,24 +2074,29 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public void commitKey(OmKeyArgs args, long clientID)
throws IOException {
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
try {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} catch (OMException ex) {
// For new keys key checkAccess call will fail as key doesn't exist.
// Check user access for bucket.
if (ex.getResult().equals(KEY_NOT_FOUND)) {
checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} else {
throw ex;
}
}
}
- Map<String, String> auditMap = (args == null) ? new LinkedHashMap<>() :
- args.toAuditMap();
+
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
auditMap.put(OzoneConsts.CLIENT_ID, String.valueOf(clientID));
+
+ args = bucket.update(args);
+
try {
metrics.incNumKeyCommits();
keyManager.commitKey(args, clientID);
@@ -2088,7 +2106,7 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
// As key also can have multiple versions, we need to increment keys
// only if version is 0. Currently we have not complete support of
// versioning of keys. So, this can be revisited later.
- if (args != null && args.getLocationInfoList() != null &&
+ if (args.getLocationInfoList() != null &&
args.getLocationInfoList().size() > 0 &&
args.getLocationInfoList().get(0) != null &&
args.getLocationInfoList().get(0).getCreateVersion() == 0) {
@@ -2105,25 +2123,30 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public OmKeyLocationInfo allocateBlock(OmKeyArgs args, long clientID,
ExcludeList excludeList) throws IOException {
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
try {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} catch (OMException ex) {
// For new keys key checkAccess call will fail as key doesn't exist.
// Check user access for bucket.
if (ex.getResult().equals(KEY_NOT_FOUND)) {
checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
} else {
throw ex;
}
}
}
+
boolean auditSuccess = true;
- Map<String, String> auditMap = (args == null) ? new LinkedHashMap<>() :
- args.toAuditMap();
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
auditMap.put(OzoneConsts.CLIENT_ID, String.valueOf(clientID));
+
+ args = bucket.update(args);
+
try {
metrics.incNumBlockAllocateCalls();
return keyManager.allocateBlock(args, clientID, excludeList);
@@ -2150,11 +2173,18 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
*/
@Override
public OmKeyInfo lookupKey(OmKeyArgs args) throws IOException {
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.READ,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
}
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumKeyLookups();
return keyManager.lookupKey(args, getClientAddress());
@@ -2162,25 +2192,32 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumKeyLookupFails();
auditSuccess = false;
AUDIT.logReadFailure(buildAuditMessageForFailure(OMAction.READ_KEY,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logReadSuccess(buildAuditMessageForSuccess(OMAction.READ_KEY,
- (args == null) ? null : args.toAuditMap()));
+ auditMap));
}
}
}
@Override
public void renameKey(OmKeyArgs args, String toKeyName) throws IOException {
+ Preconditions.checkNotNull(args);
+
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
}
- Map<String, String> auditMap = (args == null) ? new LinkedHashMap<>() :
- args.toAuditMap();
+
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
auditMap.put(OzoneConsts.TO_KEY_NAME, toKeyName);
+
+ args = bucket.update(args);
+
try {
metrics.incNumKeyRenames();
keyManager.renameKey(args, toKeyName);
@@ -2202,20 +2239,25 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
*/
@Override
public void deleteKey(OmKeyArgs args) throws IOException {
+ Map<String, String> auditMap = args.toAuditMap();
try {
+ ResolvedBucket bucket = resolveBucketLink(args);
+ args = bucket.update(args);
+
if (isAclEnabled) {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.DELETE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
}
+
metrics.incNumKeyDeletes();
keyManager.deleteKey(args);
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(OMAction.DELETE_KEY,
- (args == null) ? null : args.toAuditMap()));
+ auditMap));
metrics.decNumKeys();
} catch (Exception ex) {
metrics.incNumKeyDeleteFails();
AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction.DELETE_KEY,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
}
}
@@ -2235,19 +2277,23 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public List<OmKeyInfo> listKeys(String volumeName, String bucketName,
String startKey, String keyPrefix, int maxKeys) throws IOException {
+
+ ResolvedBucket bucket = resolveBucketLink(Pair.of(volumeName, bucketName));
+
if (isAclEnabled) {
- checkAcls(ResourceType.BUCKET,
- StoreType.OZONE, ACLType.LIST, volumeName, bucketName, keyPrefix);
+ checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.LIST,
+ bucket.realVolume(), bucket.realBucket(), keyPrefix);
}
+
boolean auditSuccess = true;
- Map<String, String> auditMap = buildAuditMap(volumeName);
- auditMap.put(OzoneConsts.BUCKET, bucketName);
+ Map<String, String> auditMap = bucket.audit();
auditMap.put(OzoneConsts.START_KEY, startKey);
auditMap.put(OzoneConsts.MAX_KEYS, String.valueOf(maxKeys));
auditMap.put(OzoneConsts.KEY_PREFIX, keyPrefix);
+
try {
metrics.incNumKeyLists();
- return keyManager.listKeys(volumeName, bucketName,
+ return keyManager.listKeys(bucket.realVolume(), bucket.realBucket(),
startKey, keyPrefix, maxKeys);
} catch (IOException ex) {
metrics.incNumKeyListFails();
@@ -2268,6 +2314,8 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
String bucketName, String startKeyName, String keyPrefix, int maxKeys)
throws IOException {
+ // bucket links not supported
+
if (isAclEnabled) {
checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.LIST,
volumeName, bucketName, keyPrefix);
@@ -2528,66 +2576,75 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public OmMultipartInfo initiateMultipartUpload(OmKeyArgs keyArgs) throws
IOException {
- OmMultipartInfo multipartInfo;
+
+ Preconditions.checkNotNull(keyArgs);
+ ResolvedBucket bucket = resolveBucketLink(keyArgs);
+
+ Map<String, String> auditMap = bucket.audit(keyArgs.toAuditMap());
+
+ keyArgs = bucket.update(keyArgs);
+
metrics.incNumInitiateMultipartUploads();
try {
- multipartInfo = keyManager.initiateMultipartUpload(keyArgs);
+ OmMultipartInfo result = keyManager.initiateMultipartUpload(keyArgs);
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
- OMAction.INITIATE_MULTIPART_UPLOAD, (keyArgs == null) ? null :
- keyArgs.toAuditMap()));
+ OMAction.INITIATE_MULTIPART_UPLOAD, auditMap));
+ return result;
} catch (IOException ex) {
AUDIT.logWriteFailure(buildAuditMessageForFailure(
- OMAction.INITIATE_MULTIPART_UPLOAD,
- (keyArgs == null) ? null : keyArgs.toAuditMap(), ex));
+ OMAction.INITIATE_MULTIPART_UPLOAD, auditMap, ex));
metrics.incNumInitiateMultipartUploadFails();
throw ex;
}
- return multipartInfo;
}
@Override
public OmMultipartCommitUploadPartInfo commitMultipartUploadPart(
OmKeyArgs keyArgs, long clientID) throws IOException {
- boolean auditSuccess = false;
- OmMultipartCommitUploadPartInfo commitUploadPartInfo;
+
+ Preconditions.checkNotNull(keyArgs);
+ ResolvedBucket bucket = resolveBucketLink(keyArgs);
+
+ Map<String, String> auditMap = bucket.audit(keyArgs.toAuditMap());
+
+ keyArgs = bucket.update(keyArgs);
+
metrics.incNumCommitMultipartUploadParts();
try {
- commitUploadPartInfo = keyManager.commitMultipartUploadPart(keyArgs,
- clientID);
- auditSuccess = true;
+ OmMultipartCommitUploadPartInfo result =
+ keyManager.commitMultipartUploadPart(keyArgs, clientID);
+ AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
+ OMAction.COMMIT_MULTIPART_UPLOAD_PARTKEY, auditMap));
+ return result;
} catch (IOException ex) {
- AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction
- .INITIATE_MULTIPART_UPLOAD, (keyArgs == null) ? null : keyArgs
- .toAuditMap(), ex));
+ AUDIT.logWriteFailure(buildAuditMessageForFailure(
+ OMAction.INITIATE_MULTIPART_UPLOAD, auditMap, ex));
metrics.incNumCommitMultipartUploadPartFails();
throw ex;
- } finally {
- if (auditSuccess) {
- AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
- OMAction.COMMIT_MULTIPART_UPLOAD_PARTKEY, (keyArgs == null) ? null :
- keyArgs.toAuditMap()));
- }
}
- return commitUploadPartInfo;
}
@Override
public OmMultipartUploadCompleteInfo completeMultipartUpload(
OmKeyArgs omKeyArgs, OmMultipartUploadCompleteList multipartUploadList)
throws IOException {
- OmMultipartUploadCompleteInfo omMultipartUploadCompleteInfo;
- metrics.incNumCompleteMultipartUploads();
- Map<String, String> auditMap = (omKeyArgs == null) ? new LinkedHashMap<>() :
- omKeyArgs.toAuditMap();
+ Preconditions.checkNotNull(omKeyArgs);
+ ResolvedBucket bucket = resolveBucketLink(omKeyArgs);
+
+ Map<String, String> auditMap = bucket.audit(omKeyArgs.toAuditMap());
auditMap.put(OzoneConsts.MULTIPART_LIST, multipartUploadList
.getMultipartMap().toString());
+
+ omKeyArgs = bucket.update(omKeyArgs);
+
+ metrics.incNumCompleteMultipartUploads();
try {
- omMultipartUploadCompleteInfo = keyManager.completeMultipartUpload(
- omKeyArgs, multipartUploadList);
+ OmMultipartUploadCompleteInfo result = keyManager.completeMultipartUpload(
+ omKeyArgs, multipartUploadList);
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(OMAction
.COMPLETE_MULTIPART_UPLOAD, auditMap));
- return omMultipartUploadCompleteInfo;
+ return result;
} catch (IOException ex) {
metrics.incNumCompleteMultipartUploadFails();
AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction
@@ -2599,8 +2656,13 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public void abortMultipartUpload(OmKeyArgs omKeyArgs) throws IOException {
- Map<String, String> auditMap = (omKeyArgs == null) ? new LinkedHashMap<>() :
- omKeyArgs.toAuditMap();
+ Preconditions.checkNotNull(omKeyArgs);
+ ResolvedBucket bucket = resolveBucketLink(omKeyArgs);
+
+ Map<String, String> auditMap = bucket.audit(omKeyArgs.toAuditMap());
+
+ omKeyArgs = bucket.update(omKeyArgs);
+
metrics.incNumAbortMultipartUploads();
try {
keyManager.abortMultipartUpload(omKeyArgs);
@@ -2616,22 +2678,24 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
}
@Override
- public OmMultipartUploadListParts listParts(String volumeName,
- String bucketName, String keyName, String uploadID, int partNumberMarker,
- int maxParts) throws IOException {
- Map<String, String> auditMap = new HashMap<>();
- auditMap.put(OzoneConsts.VOLUME, volumeName);
- auditMap.put(OzoneConsts.BUCKET, bucketName);
+ public OmMultipartUploadListParts listParts(final String volumeName,
+ final String bucketName, String keyName, String uploadID,
+ int partNumberMarker, int maxParts) throws IOException {
+
+ ResolvedBucket bucket = resolveBucketLink(Pair.of(volumeName, bucketName));
+
+ Map<String, String> auditMap = bucket.audit();
auditMap.put(OzoneConsts.KEY, keyName);
auditMap.put(OzoneConsts.UPLOAD_ID, uploadID);
auditMap.put(OzoneConsts.PART_NUMBER_MARKER,
Integer.toString(partNumberMarker));
auditMap.put(OzoneConsts.MAX_PARTS, Integer.toString(maxParts));
+
metrics.incNumListMultipartUploadParts();
try {
OmMultipartUploadListParts omMultipartUploadListParts =
- keyManager.listParts(volumeName, bucketName, keyName, uploadID,
- partNumberMarker, maxParts);
+ keyManager.listParts(bucket.realVolume(), bucket.realBucket(),
+ keyName, uploadID, partNumberMarker, maxParts);
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(OMAction
.LIST_MULTIPART_UPLOAD_PARTS, auditMap));
return omMultipartUploadListParts;
@@ -2647,15 +2711,16 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
public OmMultipartUploadList listMultipartUploads(String volumeName,
String bucketName, String prefix) throws IOException {
- Map<String, String> auditMap = new HashMap<>();
- auditMap.put(OzoneConsts.VOLUME, volumeName);
- auditMap.put(OzoneConsts.BUCKET, bucketName);
+ ResolvedBucket bucket = resolveBucketLink(Pair.of(volumeName, bucketName));
+
+ Map<String, String> auditMap = bucket.audit();
auditMap.put(OzoneConsts.PREFIX, prefix);
metrics.incNumListMultipartUploads();
try {
OmMultipartUploadList omMultipartUploadList =
- keyManager.listMultipartUploads(volumeName, bucketName, prefix);
+ keyManager.listMultipartUploads(bucket.realVolume(),
+ bucket.realBucket(), prefix);
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(OMAction
.LIST_MULTIPART_UPLOADS, auditMap));
return omMultipartUploadList;
@@ -2671,11 +2736,13 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public OzoneFileStatus getFileStatus(OmKeyArgs args) throws IOException {
- if (isAclEnabled) {
- checkAcls(getResourceType(args), StoreType.OZONE, ACLType.READ,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
- }
+ ResolvedBucket bucket = resolveBucketLink(args);
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumGetFileStatus();
return keyManager.getFileStatus(args);
@@ -2683,14 +2750,12 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumGetFileStatusFails();
auditSuccess = false;
AUDIT.logReadFailure(
- buildAuditMessageForFailure(OMAction.GET_FILE_STATUS,
- (args == null) ? null : args.toAuditMap(), ex));
+ buildAuditMessageForFailure(OMAction.GET_FILE_STATUS, auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logReadSuccess(
- buildAuditMessageForSuccess(OMAction.GET_FILE_STATUS,
- (args == null) ? null : args.toAuditMap()));
+ buildAuditMessageForSuccess(OMAction.GET_FILE_STATUS, auditMap));
}
}
}
@@ -2704,11 +2769,13 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public void createDirectory(OmKeyArgs args) throws IOException {
- if (isAclEnabled) {
- checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
- }
+ ResolvedBucket bucket = resolveBucketLink(args);
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumCreateDirectory();
keyManager.createDirectory(args);
@@ -2716,14 +2783,12 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumCreateDirectoryFails();
auditSuccess = false;
AUDIT.logWriteFailure(
- buildAuditMessageForFailure(OMAction.CREATE_DIRECTORY,
- (args == null) ? null : args.toAuditMap(), ex));
+ buildAuditMessageForFailure(OMAction.CREATE_DIRECTORY, auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logWriteSuccess(
- buildAuditMessageForSuccess(OMAction.CREATE_DIRECTORY,
- (args == null) ? null : args.toAuditMap()));
+ buildAuditMessageForSuccess(OMAction.CREATE_DIRECTORY, auditMap));
}
}
}
@@ -2731,11 +2796,13 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public OpenKeySession createFile(OmKeyArgs args, boolean overWrite,
boolean recursive) throws IOException {
- if (isAclEnabled) {
- checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.WRITE,
- args.getVolumeName(), args.getBucketName(), null);
- }
+ ResolvedBucket bucket = resolveBucketLink(args);
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumCreateFile();
return keyManager.createFile(args, overWrite, recursive);
@@ -2743,23 +2810,30 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumCreateFileFails();
auditSuccess = false;
AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction.CREATE_FILE,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
- OMAction.CREATE_FILE, (args == null) ? null : args.toAuditMap()));
+ OMAction.CREATE_FILE, auditMap));
}
}
}
@Override
public OmKeyInfo lookupFile(OmKeyArgs args) throws IOException {
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
checkAcls(ResourceType.KEY, StoreType.OZONE, ACLType.READ,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
}
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumLookupFile();
return keyManager.lookupFile(args, getClientAddress());
@@ -2767,12 +2841,12 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumLookupFileFails();
auditSuccess = false;
AUDIT.logWriteFailure(buildAuditMessageForFailure(OMAction.LOOKUP_FILE,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
- OMAction.LOOKUP_FILE, (args == null) ? null : args.toAuditMap()));
+ OMAction.LOOKUP_FILE, auditMap));
}
}
}
@@ -2780,11 +2854,19 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
@Override
public List<OzoneFileStatus> listStatus(OmKeyArgs args, boolean recursive,
String startKey, long numEntries) throws IOException {
+
+ ResolvedBucket bucket = resolveBucketLink(args);
+
if (isAclEnabled) {
checkAcls(getResourceType(args), StoreType.OZONE, ACLType.READ,
- args.getVolumeName(), args.getBucketName(), args.getKeyName());
+ bucket.realVolume(), bucket.realBucket(), args.getKeyName());
}
+
boolean auditSuccess = true;
+ Map<String, String> auditMap = bucket.audit(args.toAuditMap());
+
+ args = bucket.update(args);
+
try {
metrics.incNumListStatus();
return keyManager.listStatus(args, recursive, startKey, numEntries);
@@ -2792,12 +2874,12 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
metrics.incNumListStatusFails();
auditSuccess = false;
AUDIT.logReadFailure(buildAuditMessageForFailure(OMAction.LIST_STATUS,
- (args == null) ? null : args.toAuditMap(), ex));
+ auditMap, ex));
throw ex;
} finally {
if (auditSuccess) {
AUDIT.logReadSuccess(buildAuditMessageForSuccess(
- OMAction.LIST_STATUS, (args == null) ? null : args.toAuditMap()));
+ OMAction.LIST_STATUS, auditMap));
}
}
}
@@ -3314,4 +3396,60 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
jvmPauseMonitor.init(configuration);
jvmPauseMonitor.start();
}
+
+ public ResolvedBucket resolveBucketLink(KeyArgs args) throws IOException {
+ return resolveBucketLink(
+ Pair.of(args.getVolumeName(), args.getBucketName()));
+ }
+
+ public ResolvedBucket resolveBucketLink(OmKeyArgs args)
+ throws IOException {
+ return resolveBucketLink(
+ Pair.of(args.getVolumeName(), args.getBucketName()));
+ }
+
+ public ResolvedBucket resolveBucketLink(Pair<String, String> requested)
+ throws IOException {
+ Pair<String, String> resolved =
+ resolveBucketLink(requested, new HashSet<>());
+ return new ResolvedBucket(requested, resolved);
+ }
+
+ /**
+ * Resolves bucket symlinks. Read permission is required for following links.
+ *
+ * @param volumeAndBucket the bucket to be resolved (if it is a link)
+ * @param visited collects link buckets visited during the resolution to
+ * avoid infinite loops
+ * @return bucket location possibly updated with its actual volume and bucket
+ * after following bucket links
+ * @throws IOException (most likely OMException) if ACL check fails, bucket is
+ * not found, loop is detected in the links, etc.
+ */
+ private Pair<String, String> resolveBucketLink(
+ Pair<String, String> volumeAndBucket,
+ Set<Pair<String, String>> visited) throws IOException {
+
+ String volumeName = volumeAndBucket.getLeft();
+ String bucketName = volumeAndBucket.getRight();
+ OmBucketInfo info = bucketManager.getBucketInfo(volumeName, bucketName);
+ if (!info.isLink()) {
+ return volumeAndBucket;
+ }
+
+ if (!visited.add(volumeAndBucket)) {
+ throw new OMException("Detected loop in bucket links",
+ DETECTED_LOOP_IN_BUCKET_LINKS);
+ }
+
+ if (isAclEnabled) {
+ checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.READ,
+ volumeName, bucketName, null);
+ }
+
+ return resolveBucketLink(
+ Pair.of(info.getSourceVolume(), info.getSourceBucket()),
+ visited);
+ }
+
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java
new file mode 100644
index 0000000..fef9b2e
--- /dev/null
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.helpers.OmKeyArgs;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Bundles information about a bucket, which is possibly a symlink,
+ * and the real bucket that it resolves to, if it is indeed a link.
+ * For regular buckets, both {@code requested} and {@code resolved} point to
+ * the same bucket.
+ */
+public class ResolvedBucket {
+
+ private final Pair<String, String> requested;
+ private final Pair<String, String> resolved;
+
+ public ResolvedBucket(Pair<String, String> requested,
+ Pair<String, String> resolved) {
+ this.requested = requested;
+ this.resolved = resolved;
+ }
+
+ public Pair<String, String> requested() {
+ return requested;
+ }
+
+ public Pair<String, String> resolved() {
+ return resolved;
+ }
+
+ public String requestedVolume() {
+ return requested.getLeft();
+ }
+
+ public String requestedBucket() {
+ return requested.getRight();
+ }
+
+ public String realVolume() {
+ return resolved.getLeft();
+ }
+
+ public String realBucket() {
+ return resolved.getRight();
+ }
+
+ public OmKeyArgs update(OmKeyArgs args) {
+ return isLink()
+ ? args.toBuilder()
+ .setVolumeName(realVolume())
+ .setBucketName(realBucket())
+ .build()
+ : args;
+ }
+
+ public KeyArgs update(KeyArgs args) {
+ return isLink()
+ ? args.toBuilder()
+ .setVolumeName(realVolume())
+ .setBucketName(realBucket())
+ .build()
+ : args;
+ }
+
+ public boolean isLink() {
+ return !Objects.equals(requested, resolved);
+ }
+
+ public Map<String, String> audit() {
+ return audit(new LinkedHashMap<>());
+ }
+
+ /**
+ * Adds audit information about the bucket (and if it's a link, then the
+ * real bucket, too) to {@code auditMap}.
+ * @return the same map for convenience
+ */
+ public Map<String, String> audit(Map<String, String> auditMap) {
+ auditMap.putIfAbsent(OzoneConsts.VOLUME, requestedVolume());
+ auditMap.putIfAbsent(OzoneConsts.BUCKET, requestedBucket());
+ if (isLink()) {
+ auditMap.put(OzoneConsts.SOURCE_VOLUME, realVolume());
+ auditMap.put(OzoneConsts.SOURCE_BUCKET, realBucket());
+ }
+ return auditMap;
+ }
+
+}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
index 9d7d133..71d5458 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
@@ -115,6 +115,20 @@ public class OMBucketCreateRequest extends OMClientRequest {
newBucketInfo.setBeinfo(getBeinfo(kmsProvider, bucketInfo));
}
+ boolean hasSourceVolume = bucketInfo.getSourceVolume() != null;
+ boolean hasSourceBucket = bucketInfo.getSourceBucket() != null;
+
+ if (hasSourceBucket != hasSourceVolume) {
+ throw new OMException("Both source volume and source bucket are " +
+ "required for bucket links",
+ OMException.ResultCodes.INVALID_REQUEST);
+ }
+
+ if (hasSourceBucket && bucketInfo.hasBeinfo()) {
+ throw new OMException("Encryption cannot be set for bucket links",
+ OMException.ResultCodes.INVALID_REQUEST);
+ }
+
newCreateBucketRequest.setBucketInfo(newBucketInfo.build());
return getOmRequest().toBuilder().setUserInfo(getUserInfo())
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
index ec51333..7b2ab51 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
@@ -146,6 +146,10 @@ public class OMDirectoryCreateRequest extends OMKeyRequest {
List<OmKeyInfo> missingParentInfos;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acl
checkKeyAcls(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.CREATE, OzoneObj.ResourceType.KEY);
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
index 3b0b02b..7327626 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
@@ -166,6 +166,7 @@ public class OMFileCreateRequest extends OMKeyRequest {
CreateFileRequest createFileRequest = getOmRequest().getCreateFileRequest();
KeyArgs keyArgs = createFileRequest.getKeyArgs();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
@@ -199,6 +200,10 @@ public class OMFileCreateRequest extends OMKeyRequest {
IOException exception = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acl
checkKeyAcls(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.CREATE, OzoneObj.ResourceType.KEY);
@@ -310,7 +315,6 @@ public class OMFileCreateRequest extends OMKeyRequest {
}
// Audit Log outside the lock
- Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
auditLog(ozoneManager.getAuditLogger(), buildAuditMessage(
OMAction.CREATE_FILE, auditMap, exception,
getOmRequest().getUserInfo()));
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
index 1a39e0b..9e82888 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
@@ -161,8 +161,7 @@ public class OMAllocateBlockRequest extends OMKeyRequest {
auditMap.put(OzoneConsts.CLIENT_ID, String.valueOf(clientID));
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
- String openKeyName = omMetadataManager.getOpenKey(volumeName, bucketName,
- keyName, clientID);
+ String openKeyName = null;
OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
getOmRequest());
@@ -172,6 +171,10 @@ public class OMAllocateBlockRequest extends OMKeyRequest {
IOException exception = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acl
checkKeyAclsInOpenKeyTable(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.WRITE, allocateBlockRequest.getClientID());
@@ -182,6 +185,8 @@ public class OMAllocateBlockRequest extends OMKeyRequest {
// Here we don't acquire bucket/volume lock because for a single client
// allocateBlock is called in serial fashion.
+ openKeyName = omMetadataManager.getOpenKey(volumeName, bucketName,
+ keyName, clientID);
openKeyInfo = omMetadataManager.getOpenKeyTable().get(openKeyName);
if (openKeyInfo == null) {
throw new OMException("Open Key not found " + openKeyName,
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
index edeea3d..eb3769b 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
@@ -125,37 +125,42 @@ public class OMKeyCommitRequest extends OMKeyRequest {
OmKeyInfo omKeyInfo = null;
OMClientResponse omClientResponse = null;
boolean bucketLockAcquired = false;
- Result result = null;
+ Result result;
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
- String dbOzoneKey = omMetadataManager.getOzoneKey(volumeName, bucketName,
- keyName);
- String dbOpenKey = omMetadataManager.getOpenKey(volumeName, bucketName,
- keyName, commitKeyRequest.getClientID());
try {
+ commitKeyArgs = resolveBucketLink(ozoneManager, commitKeyArgs, auditMap);
+ volumeName = commitKeyArgs.getVolumeName();
+ bucketName = commitKeyArgs.getBucketName();
+
// check Acl
checkKeyAclsInOpenKeyTable(ozoneManager, volumeName, bucketName,
keyName, IAccessAuthorizer.ACLType.WRITE,
commitKeyRequest.getClientID());
+ String dbOzoneKey =
+ omMetadataManager.getOzoneKey(volumeName, bucketName,
+ keyName);
+ String dbOpenKey = omMetadataManager.getOpenKey(volumeName, bucketName,
+ keyName, commitKeyRequest.getClientID());
+
List<OmKeyLocationInfo> locationInfoList = new ArrayList<>();
for (KeyLocation keyLocation : commitKeyArgs.getKeyLocationsList()) {
locationInfoList.add(OmKeyLocationInfo.getFromProtobuf(keyLocation));
}
- bucketLockAcquired = omMetadataManager.getLock().acquireWriteLock(
- BUCKET_LOCK, volumeName, bucketName);
+ bucketLockAcquired =
+ omMetadataManager.getLock().acquireLock(BUCKET_LOCK,
+ volumeName, bucketName);
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
omKeyInfo = omMetadataManager.getOpenKeyTable().get(dbOpenKey);
-
if (omKeyInfo == null) {
throw new OMException("Failed to commit key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
-
omKeyInfo.setDataSize(commitKeyArgs.getDataSize());
omKeyInfo.setModificationTime(commitKeyArgs.getModificationTime());
@@ -183,7 +188,7 @@ public class OMKeyCommitRequest extends OMKeyRequest {
result = Result.FAILURE;
exception = ex;
omClientResponse = new OMKeyCommitResponse(createErrorOMResponse(
- omResponse, exception));
+ omResponse, exception));
} finally {
addResponseToDoubleBuffer(trxnLogIndex, omClientResponse,
omDoubleBufferHelper);
@@ -207,7 +212,7 @@ public class OMKeyCommitRequest extends OMKeyRequest {
if (omKeyInfo.getKeyLocationVersions().size() == 1) {
omMetrics.incNumKeys();
}
- LOG.debug("Key commited. Volume:{}, Bucket:{}, Key:{}", volumeName,
+ LOG.debug("Key committed. Volume:{}, Bucket:{}, Key:{}", volumeName,
bucketName, keyName);
break;
case FAILURE:
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
index c6a7e52..f7f08dc 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
@@ -165,6 +165,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
CreateKeyRequest createKeyRequest = getOmRequest().getCreateKeyRequest();
KeyArgs keyArgs = createKeyRequest.getKeyArgs();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
@@ -184,6 +185,10 @@ public class OMKeyCreateRequest extends OMKeyRequest {
IOException exception = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acl
checkKeyAcls(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.CREATE, OzoneObj.ResourceType.KEY);
@@ -253,13 +258,10 @@ public class OMKeyCreateRequest extends OMKeyRequest {
}
// Audit Log outside the lock
-
- Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
auditLog(ozoneManager.getAuditLogger(), buildAuditMessage(
OMAction.ALLOCATE_KEY, auditMap, exception,
getOmRequest().getUserInfo()));
-
switch (result) {
case SUCCESS:
LOG.debug("Key created. Volume:{}, Bucket:{}, Key:{}", volumeName,
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
index b0eb6fd..8b75417 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
@@ -88,12 +88,13 @@ public class OMKeyDeleteRequest extends OMKeyRequest {
long trxnLogIndex, OzoneManagerDoubleBufferHelper omDoubleBufferHelper) {
DeleteKeyRequest deleteKeyRequest = getOmRequest().getDeleteKeyRequest();
- OzoneManagerProtocolProtos.KeyArgs deleteKeyArgs =
+ OzoneManagerProtocolProtos.KeyArgs keyArgs =
deleteKeyRequest.getKeyArgs();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
- String volumeName = deleteKeyArgs.getVolumeName();
- String bucketName = deleteKeyArgs.getBucketName();
- String keyName = deleteKeyArgs.getKeyName();
+ String volumeName = keyArgs.getVolumeName();
+ String bucketName = keyArgs.getBucketName();
+ String keyName = keyArgs.getKeyName();
OMMetrics omMetrics = ozoneManager.getMetrics();
omMetrics.incNumKeyDeletes();
@@ -101,8 +102,6 @@ public class OMKeyDeleteRequest extends OMKeyRequest {
AuditLogger auditLogger = ozoneManager.getAuditLogger();
OzoneManagerProtocolProtos.UserInfo userInfo = getOmRequest().getUserInfo();
- Map<String, String> auditMap = buildKeyArgsAuditMap(deleteKeyArgs);
-
OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
getOmRequest());
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
@@ -111,6 +110,10 @@ public class OMKeyDeleteRequest extends OMKeyRequest {
OMClientResponse omClientResponse = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acl
checkKeyAcls(ozoneManager, volumeName, bucketName, keyName,
IAccessAuthorizer.ACLType.DELETE, OzoneObj.ResourceType.KEY);
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
index dc83ff6..91db347 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
@@ -101,12 +101,13 @@ public class OMKeyRenameRequest extends OMKeyRequest {
long trxnLogIndex, OzoneManagerDoubleBufferHelper omDoubleBufferHelper) {
RenameKeyRequest renameKeyRequest = getOmRequest().getRenameKeyRequest();
- OzoneManagerProtocolProtos.KeyArgs renameKeyArgs =
+ OzoneManagerProtocolProtos.KeyArgs keyArgs =
renameKeyRequest.getKeyArgs();
+ Map<String, String> auditMap = buildAuditMap(keyArgs, renameKeyRequest);
- String volumeName = renameKeyArgs.getVolumeName();
- String bucketName = renameKeyArgs.getBucketName();
- String fromKeyName = renameKeyArgs.getKeyName();
+ String volumeName = keyArgs.getVolumeName();
+ String bucketName = keyArgs.getBucketName();
+ String fromKeyName = keyArgs.getKeyName();
String toKeyName = renameKeyRequest.getToKeyName();
OMMetrics omMetrics = ozoneManager.getMetrics();
@@ -114,9 +115,6 @@ public class OMKeyRenameRequest extends OMKeyRequest {
AuditLogger auditLogger = ozoneManager.getAuditLogger();
- Map<String, String> auditMap =
- buildAuditMap(renameKeyArgs, renameKeyRequest);
-
OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
getOmRequest());
@@ -132,6 +130,11 @@ public class OMKeyRenameRequest extends OMKeyRequest {
throw new OMException("Key name is empty",
OMException.ResultCodes.INVALID_KEY_NAME);
}
+
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// check Acls to see if user has access to perform delete operation on
// old key and create operation on new key
checkKeyAcls(ozoneManager, volumeName, bucketName, fromKeyName,
@@ -168,7 +171,7 @@ public class OMKeyRenameRequest extends OMKeyRequest {
fromKeyValue.setKeyName(toKeyName);
//Set modification time
- fromKeyValue.setModificationTime(renameKeyArgs.getModificationTime());
+ fromKeyValue.setModificationTime(keyArgs.getModificationTime());
// Add to cache.
// fromKey should be deleted, toKey should be added with newly updated
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java
index 0aec04d..e3f0a69 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java
@@ -27,11 +27,13 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Map;
import com.google.common.base.Optional;
import com.google.common.base.Preconditions;
import org.apache.hadoop.ozone.OzoneAcl;
import org.apache.hadoop.ozone.om.PrefixManager;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
import org.apache.hadoop.ozone.om.helpers.BucketEncryptionKeyInfo;
import org.apache.hadoop.ozone.om.helpers.KeyValueUtil;
import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
@@ -88,6 +90,15 @@ public abstract class OMKeyRequest extends OMClientRequest {
super(omRequest);
}
+ protected static KeyArgs resolveBucketLink(
+ OzoneManager ozoneManager, KeyArgs keyArgs,
+ Map<String, String> auditMap) throws IOException {
+ ResolvedBucket bucket = ozoneManager.resolveBucketLink(keyArgs);
+ keyArgs = bucket.update(keyArgs);
+ bucket.audit(auditMap);
+ return keyArgs;
+ }
+
/**
* This methods avoids multiple rpc calls to SCM by allocating multiple blocks
* in one rpc call.
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java
index adc42d8..012df49 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java
@@ -19,12 +19,14 @@
package org.apache.hadoop.ozone.om.request.key;
import com.google.common.base.Optional;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
import org.apache.hadoop.ozone.audit.AuditLogger;
import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OMMetrics;
import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
import org.apache.hadoop.ozone.om.request.util.OmResponseUtil;
@@ -42,7 +44,7 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -85,10 +87,11 @@ public class OMKeysDeleteRequest extends OMKeyRequest {
OMMetrics omMetrics = ozoneManager.getMetrics();
omMetrics.incNumKeyDeletes();
- Map<String, String> auditMap = null;
String volumeName = deleteKeyArgs.getVolumeName();
String bucketName = deleteKeyArgs.getBucketName();
- String keyName = "";
+ Map<String, String> auditMap = new LinkedHashMap<>();
+ auditMap.put(VOLUME, volumeName);
+ auditMap.put(BUCKET, bucketName);
List<OmKeyInfo> omKeyInfoList = new ArrayList<>();
AuditLogger auditLogger = ozoneManager.getAuditLogger();
@@ -99,10 +102,7 @@ public class OMKeysDeleteRequest extends OMKeyRequest {
getOmRequest());
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
-
- boolean acquiredLock =
- omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ boolean acquiredLock = false;
int indexFailed = 0;
int length = deleteKeys.size();
@@ -112,12 +112,19 @@ public class OMKeysDeleteRequest extends OMKeyRequest {
boolean deleteStatus = true;
try {
-
+ ResolvedBucket bucket = ozoneManager.resolveBucketLink(
+ Pair.of(volumeName, bucketName));
+ bucket.audit(auditMap);
+ volumeName = bucket.realVolume();
+ bucketName = bucket.realBucket();
+
+ acquiredLock = omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
// Validate bucket and volume exists or not.
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
for (indexFailed = 0; indexFailed < length; indexFailed++) {
- keyName = deleteKeyArgs.getKeys(indexFailed);
+ String keyName = deleteKeyArgs.getKeys(indexFailed);
String objectKey = omMetadataManager.getOzoneKey(volumeName, bucketName,
keyName);
OmKeyInfo omKeyInfo = omMetadataManager.getKeyTable().get(objectKey);
@@ -187,8 +194,7 @@ public class OMKeysDeleteRequest extends OMKeyRequest {
omDoubleBufferHelper);
}
- auditMap = buildDeleteKeysAuditMap(volumeName, bucketName, deleteKeys,
- unDeletedKeys.getKeysList());
+ addDeletedKeys(auditMap, deleteKeys, unDeletedKeys.getKeysList());
auditLog(auditLogger, buildAuditMessage(DELETE_KEYS, auditMap, exception,
userInfo));
@@ -221,21 +227,13 @@ public class OMKeysDeleteRequest extends OMKeyRequest {
}
/**
- * Build audit map for DeleteKeys request.
- * @param volumeName
- * @param bucketName
- * @param deletedKeys
- * @param unDeletedKeys
- * @return
+ * Add key info to audit map for DeleteKeys request.
*/
- private Map<String, String> buildDeleteKeysAuditMap(String volumeName,
- String bucketName, List<String> deletedKeys, List<String> unDeletedKeys) {
- Map< String, String > auditMap = new HashMap<>();
- auditMap.put(VOLUME, volumeName);
- auditMap.put(BUCKET, bucketName);
+ private static void addDeletedKeys(
+ Map<String, String> auditMap, List<String> deletedKeys,
+ List<String> unDeletedKeys) {
auditMap.put(DELETED_KEYS_LIST, String.join(",", deletedKeys));
- auditMap.put(UNDELETED_KEYS_LIST, String.join(",",
- unDeletedKeys));
- return auditMap;
+ auditMap.put(UNDELETED_KEYS_LIST, String.join(",", unDeletedKeys));
}
+
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMTrashRecoverRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMTrashRecoverRequest.java
index eac7842..232a0fb 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMTrashRecoverRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMTrashRecoverRequest.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.ozone.om.request.key;
import java.io.IOException;
import com.google.common.base.Preconditions;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
import org.apache.hadoop.ozone.om.response.key.OMTrashRecoverResponse;
import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
@@ -86,6 +88,11 @@ public class OMTrashRecoverRequest extends OMKeyRequest {
boolean acquireLock = false;
OMClientResponse omClientResponse = null;
try {
+ ResolvedBucket bucket = ozoneManager.resolveBucketLink(
+ Pair.of(volumeName, destinationBucket));
+ volumeName = bucket.realVolume();
+ destinationBucket = bucket.realBucket();
+
// Check acl for the destination bucket.
checkBucketAcls(ozoneManager, volumeName, destinationBucket, keyName,
IAccessAuthorizer.ACLType.WRITE);
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
index 4f95fe4..aa96ba9 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
@@ -48,6 +48,7 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.Map;
import java.util.UUID;
import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
@@ -96,8 +97,12 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
Preconditions.checkNotNull(keyArgs.getMultipartUploadID());
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
+
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
+ final String requestedVolume = volumeName;
+ final String requestedBucket = bucketName;
String keyName = keyArgs.getKeyName();
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
@@ -114,10 +119,14 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
getOmRequest());
OMClientResponse omClientResponse = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// TODO to support S3 ACL later.
acquiredBucketLock =
- omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
@@ -136,8 +145,9 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
// multipart upload request is received, it returns multipart upload id
// for the key.
- String multipartKey = omMetadataManager.getMultipartKey(volumeName,
- bucketName, keyName, keyArgs.getMultipartUploadID());
+ String multipartKey = omMetadataManager.getMultipartKey(
+ volumeName, bucketName, keyName,
+ keyArgs.getMultipartUploadID());
// Even if this key already exists in the KeyTable, it would be taken
// care of in the final complete multipart upload. AWS S3 behavior is
@@ -154,8 +164,8 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
.build();
omKeyInfo = new OmKeyInfo.Builder()
- .setVolumeName(keyArgs.getVolumeName())
- .setBucketName(keyArgs.getBucketName())
+ .setVolumeName(volumeName)
+ .setBucketName(bucketName)
.setKeyName(keyArgs.getKeyName())
.setCreationTime(keyArgs.getModificationTime())
.setModificationTime(keyArgs.getModificationTime())
@@ -180,8 +190,8 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
new S3InitiateMultipartUploadResponse(
omResponse.setInitiateMultiPartUploadResponse(
MultipartInfoInitiateResponse.newBuilder()
- .setVolumeName(volumeName)
- .setBucketName(bucketName)
+ .setVolumeName(requestedVolume)
+ .setBucketName(requestedBucket)
.setKeyName(keyName)
.setMultipartUploadID(keyArgs.getMultipartUploadID()))
.build(), multipartKeyInfo, omKeyInfo);
@@ -196,14 +206,14 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
addResponseToDoubleBuffer(transactionLogIndex, omClientResponse,
ozoneManagerDoubleBufferHelper);
if (acquiredBucketLock) {
- omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
}
}
// audit log
auditLog(ozoneManager.getAuditLogger(), buildAuditMessage(
- OMAction.INITIATE_MULTIPART_UPLOAD, buildKeyArgsAuditMap(keyArgs),
+ OMAction.INITIATE_MULTIPART_UPLOAD, auditMap,
exception, getOmRequest().getUserInfo()));
switch (result) {
@@ -217,6 +227,7 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
LOG.error("S3 InitiateMultipart Upload request for Key {} in " +
"Volume/Bucket {}/{} is failed", keyName, volumeName, bucketName,
exception);
+ break;
default:
LOG.error("Unrecognized Result for S3InitiateMultipartUploadRequest: {}",
multipartInfoInitiateRequest);
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
index 4518a3b..0726fe4 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.ozone.om.request.s3.multipart;
import java.io.IOException;
+import java.util.Map;
import com.google.common.base.Optional;
import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
@@ -85,9 +86,12 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
.getAbortMultiPartUploadRequest();
OzoneManagerProtocolProtos.KeyArgs keyArgs = multipartUploadAbortRequest
.getKeyArgs();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
+ final String requestedVolume = volumeName;
+ final String requestedBucket = bucketName;
String keyName = keyArgs.getKeyName();
ozoneManager.getMetrics().incNumAbortMultipartUploads();
@@ -101,15 +105,19 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
OMClientResponse omClientResponse = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// TODO to support S3 ACL later.
acquiredLock =
- omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
- multipartKey = omMetadataManager.getMultipartKey(volumeName,
- bucketName, keyName, keyArgs.getMultipartUploadID());
+ multipartKey = omMetadataManager.getMultipartKey(
+ volumeName, bucketName, keyName, keyArgs.getMultipartUploadID());
OmKeyInfo omKeyInfo =
omMetadataManager.getOpenKeyTable().get(multipartKey);
@@ -118,7 +126,7 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
// upload initiated for this key.
if (omKeyInfo == null) {
throw new OMException("Abort Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName,
+ requestedVolume + "bucket: " + requestedBucket + "key: " + keyName,
OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
}
@@ -152,14 +160,14 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
addResponseToDoubleBuffer(trxnLogIndex, omClientResponse,
omDoubleBufferHelper);
if (acquiredLock) {
- omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
}
}
// audit log
auditLog(ozoneManager.getAuditLogger(), buildAuditMessage(
- OMAction.ABORT_MULTIPART_UPLOAD, buildKeyArgsAuditMap(keyArgs),
+ OMAction.ABORT_MULTIPART_UPLOAD, auditMap,
exception, getOmRequest().getUserInfo()));
switch (result) {
@@ -173,6 +181,7 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
LOG.error("Abort Multipart request is failed for KeyName {} in " +
"VolumeName/Bucket {}/{}", keyName, volumeName, bucketName,
exception);
+ break;
default:
LOG.error("Unrecognized Result for S3MultipartUploadAbortRequest: {}",
multipartUploadAbortRequest);
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
index 346ff87..283a22d 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
@@ -89,6 +89,7 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
getOmRequest().getCommitMultiPartUploadRequest();
KeyArgs keyArgs = multipartCommitUploadPartRequest.getKeyArgs();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
@@ -111,6 +112,10 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
OmMultipartKeyInfo multipartKeyInfo = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
// TODO to support S3 ACL later.
acquiredLock = omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK,
volumeName, bucketName);
@@ -118,16 +123,19 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
String uploadID = keyArgs.getMultipartUploadID();
- multipartKey = omMetadataManager.getMultipartKey(volumeName,
- bucketName, keyName, uploadID);
+ multipartKey = omMetadataManager.getMultipartKey(volumeName, bucketName,
+ keyName, uploadID);
multipartKeyInfo = omMetadataManager.getMultipartInfoTable()
.get(multipartKey);
long clientID = multipartCommitUploadPartRequest.getClientID();
- openKey = omMetadataManager.getOpenKey(volumeName, bucketName, keyName,
- clientID);
+ openKey = omMetadataManager.getOpenKey(
+ volumeName, bucketName, keyName, clientID);
+
+ String ozoneKey = omMetadataManager.getOzoneKey(
+ volumeName, bucketName, keyName);
omKeyInfo = omMetadataManager.getOpenKeyTable().get(openKey);
@@ -147,8 +155,7 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
// Set the UpdateID to current transactionLogIndex
omKeyInfo.setUpdateID(trxnLogIndex, ozoneManager.isRatisEnabled());
- partName = omMetadataManager.getOzoneKey(volumeName, bucketName,
- keyName) + clientID;
+ partName = ozoneKey + clientID;
if (multipartKeyInfo == null) {
// This can occur when user started uploading part by the time commit
@@ -217,15 +224,19 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
addResponseToDoubleBuffer(trxnLogIndex, omClientResponse,
omDoubleBufferHelper);
if (acquiredLock) {
- omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
}
}
// audit log
+ // Add MPU related information.
+ auditMap.put(OzoneConsts.MULTIPART_UPLOAD_PART_NUMBER,
+ String.valueOf(keyArgs.getMultipartNumber()));
+ auditMap.put(OzoneConsts.MULTIPART_UPLOAD_PART_NAME, partName);
auditLog(ozoneManager.getAuditLogger(), buildAuditMessage(
OMAction.COMMIT_MULTIPART_UPLOAD_PARTKEY,
- buildAuditMap(keyArgs, partName), exception,
+ auditMap, exception,
getOmRequest().getUserInfo()));
switch (result) {
@@ -236,7 +247,8 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
case FAILURE:
ozoneManager.getMetrics().incNumCommitMultipartUploadPartFails();
LOG.error("MultipartUpload Commit is failed for Key:{} in " +
- "Volume/Bucket {}/{}", keyName, volumeName, bucketName, exception);
+ "Volume/Bucket {}/{}", keyName, volumeName, bucketName,
+ exception);
break;
default:
LOG.error("Unrecognized Result for S3MultipartUploadCommitPartRequest: " +
@@ -246,15 +258,5 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
return omClientResponse;
}
- private Map<String, String> buildAuditMap(KeyArgs keyArgs, String partName) {
- Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
-
- // Add MPU related information.
- auditMap.put(OzoneConsts.MULTIPART_UPLOAD_PART_NUMBER,
- String.valueOf(keyArgs.getMultipartNumber()));
- auditMap.put(OzoneConsts.MULTIPART_UPLOAD_PART_NAME, partName);
-
- return auditMap;
- }
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
index c4e315c..a9aefa0 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
@@ -96,19 +96,19 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
List<OzoneManagerProtocolProtos.Part> partsList =
multipartUploadCompleteRequest.getPartsListList();
+ Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
String volumeName = keyArgs.getVolumeName();
String bucketName = keyArgs.getBucketName();
+ final String requestedVolume = volumeName;
+ final String requestedBucket = bucketName;
String keyName = keyArgs.getKeyName();
String uploadID = keyArgs.getMultipartUploadID();
+ String multipartKey = null;
ozoneManager.getMetrics().incNumCompleteMultipartUploads();
OMMetadataManager omMetadataManager = ozoneManager.getMetadataManager();
- String multipartKey = omMetadataManager.getMultipartKey(volumeName,
- bucketName, keyName, uploadID);
- String ozoneKey = omMetadataManager.getOzoneKey(volumeName, bucketName,
- keyName);
boolean acquiredLock = false;
OMResponse.Builder omResponse = OmResponseUtil.getOMResponseBuilder(
@@ -117,6 +117,13 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
IOException exception = null;
Result result = null;
try {
+ keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
+ volumeName = keyArgs.getVolumeName();
+ bucketName = keyArgs.getBucketName();
+
+ multipartKey = omMetadataManager.getMultipartKey(volumeName,
+ bucketName, keyName, uploadID);
+
// TODO to support S3 ACL later.
acquiredLock = omMetadataManager.getLock().acquireWriteLock(BUCKET_LOCK,
@@ -124,12 +131,15 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
validateBucketAndVolume(omMetadataManager, volumeName, bucketName);
+ String ozoneKey = omMetadataManager.getOzoneKey(
+ volumeName, bucketName, keyName);
+
OmMultipartKeyInfo multipartKeyInfo = omMetadataManager
.getMultipartInfoTable().get(multipartKey);
if (multipartKeyInfo == null) {
- throw new OMException("Complete Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName,
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName),
OMException.ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR);
}
TreeMap<Integer, PartKeyInfo> partKeyInfoMap =
@@ -140,8 +150,8 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
LOG.error("Complete MultipartUpload failed for key {} , MPU Key has" +
" no parts in OM, parts given to upload are {}", ozoneKey,
partsList);
- throw new OMException("Complete Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName,
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName),
OMException.ResultCodes.INVALID_PART);
}
@@ -157,9 +167,9 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
"partNumber at index {} is {} for ozonekey is " +
"{}", i, currentPartNumber, i - 1, prevPartNumber,
ozoneKey);
- throw new OMException("Complete Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName +
- "because parts are in Invalid order.",
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName) +
+ " because parts are in Invalid order.",
OMException.ResultCodes.INVALID_PART_ORDER);
}
prevPartNumber = currentPartNumber;
@@ -182,10 +192,10 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
!partName.equals(partKeyInfo.getPartName())) {
String omPartName = partKeyInfo == null ? null :
partKeyInfo.getPartName();
- throw new OMException("Complete Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName +
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName) +
". Provided Part info is { " + partName + ", " + partNumber +
- "}, where as OM has partName " + omPartName,
+ "}, whereas OM has partName " + omPartName,
OMException.ResultCodes.INVALID_PART);
}
@@ -200,9 +210,9 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
partKeyInfo.getPartNumber(),
currentPartKeyInfo.getDataSize(),
OzoneConsts.OM_MULTIPART_MIN_SIZE);
- throw new OMException("Complete Multipart Upload Failed: " +
- "Entity too small: volume: " + volumeName + "bucket: " +
- bucketName + "key: " + keyName,
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName) +
+ ". Entity too small.",
OMException.ResultCodes.ENTITY_TOO_SMALL);
}
}
@@ -275,8 +285,8 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
omResponse.setCompleteMultiPartUploadResponse(
MultipartUploadCompleteResponse.newBuilder()
- .setVolume(volumeName)
- .setBucket(bucketName)
+ .setVolume(requestedVolume)
+ .setBucket(requestedBucket)
.setKey(keyName)
.setHash(DigestUtils.sha256Hex(keyName)));
@@ -285,9 +295,9 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
result = Result.SUCCESS;
} else {
- throw new OMException("Complete Multipart Upload Failed: volume: " +
- volumeName + "bucket: " + bucketName + "key: " + keyName +
- "because of empty part list",
+ throw new OMException(
+ failureMessage(requestedVolume, requestedBucket, keyName) +
+ " because of empty part list",
OMException.ResultCodes.INVALID_REQUEST);
}
@@ -300,12 +310,11 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
addResponseToDoubleBuffer(trxnLogIndex, omClientResponse,
omDoubleBufferHelper);
if (acquiredLock) {
- omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK, volumeName,
- bucketName);
+ omMetadataManager.getLock().releaseWriteLock(BUCKET_LOCK,
+ volumeName, bucketName);
}
}
- Map<String, String> auditMap = buildKeyArgsAuditMap(keyArgs);
auditMap.put(OzoneConsts.MULTIPART_LIST, partsList.toString());
// audit log
@@ -315,13 +324,15 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
switch (result) {
case SUCCESS:
- LOG.debug("MultipartUpload Complete request is successfull for Key: {} " +
+ LOG.debug("MultipartUpload Complete request is successful for Key: {} " +
"in Volume/Bucket {}/{}", keyName, volumeName, bucketName);
break;
case FAILURE:
ozoneManager.getMetrics().incNumCompleteMultipartUploadFails();
LOG.error("MultipartUpload Complete request failed for Key: {} " +
- "in Volume/Bucket {}/{}", keyName, volumeName, bucketName, exception);
+ "in Volume/Bucket {}/{}", keyName, volumeName, bucketName,
+ exception);
+ break;
default:
LOG.error("Unrecognized Result for S3MultipartUploadCommitRequest: {}",
multipartUploadCompleteRequest);
@@ -330,6 +341,12 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
return omClientResponse;
}
+ private static String failureMessage(String volume, String bucket,
+ String keyName) {
+ return "Complete Multipart Upload Failed: volume: " +
+ volume + " bucket: " + bucket + " key: " + keyName;
+ }
+
private void updateCache(OMMetadataManager omMetadataManager,
String ozoneKey, String multipartKey, OmKeyInfo omKeyInfo,
long transactionLogIndex) {
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java
index b714375..c09bf86 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java
@@ -21,7 +21,9 @@ package org.apache.hadoop.ozone.om.request.file;
import java.util.UUID;
import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
import org.apache.hadoop.ozone.om.helpers.OzoneFSUtils;
import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
import org.apache.hadoop.ozone.om.response.OMClientResponse;
@@ -86,6 +88,8 @@ public class TestOMDirectoryCreateRequest {
auditLogger = Mockito.mock(AuditLogger.class);
when(ozoneManager.getAuditLogger()).thenReturn(auditLogger);
Mockito.doNothing().when(auditLogger).logWrite(any(AuditMessage.class));
+ when(ozoneManager.resolveBucketLink(any(KeyArgs.class)))
+ .thenReturn(new ResolvedBucket(Pair.of("", ""), Pair.of("", "")));
}
@After
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRequest.java
index 49794a1..dd6caf4 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRequest.java
@@ -22,7 +22,10 @@ import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
import org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerDoubleBufferHelper;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
@@ -150,6 +153,11 @@ public class TestOMKeyRequest {
clientID = Time.now();
dataSize = 1000L;
+ Pair<String, String> volumeAndBucket = Pair.of(volumeName, bucketName);
+ when(ozoneManager.resolveBucketLink(any(KeyArgs.class)))
+ .thenReturn(new ResolvedBucket(volumeAndBucket, volumeAndBucket));
+ when(ozoneManager.resolveBucketLink(any(Pair.class)))
+ .thenReturn(new ResolvedBucket(volumeAndBucket, volumeAndBucket));
}
@After
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
index 9950027..0271a7a 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
@@ -29,6 +29,7 @@ import org.junit.Rule;
import org.junit.rules.TemporaryFolder;
import org.mockito.Mockito;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.audit.AuditLogger;
import org.apache.hadoop.ozone.audit.AuditMessage;
@@ -37,6 +38,8 @@ import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OMMetrics;
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.ResolvedBucket;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
.OMRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Part;
@@ -79,6 +82,13 @@ public class TestS3MultipartRequest {
auditLogger = Mockito.mock(AuditLogger.class);
when(ozoneManager.getAuditLogger()).thenReturn(auditLogger);
Mockito.doNothing().when(auditLogger).logWrite(any(AuditMessage.class));
+ when(ozoneManager.resolveBucketLink(any(KeyArgs.class)))
+ .thenAnswer(inv -> {
+ KeyArgs args = (KeyArgs) inv.getArguments()[0];
+ return new ResolvedBucket(
+ Pair.of(args.getVolumeName(), args.getBucketName()),
+ Pair.of(args.getVolumeName(), args.getBucketName()));
+ });
}
diff --git a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
index e723df7..25dea3d 100644
--- a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
+++ b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
@@ -477,9 +477,7 @@ public class BasicOzoneFileSystem extends FileSystem {
result = innerDelete(f, recursive);
} else {
LOG.debug("delete: Path is a file: {}", f);
- List<String> keyList = new ArrayList<>();
- keyList.add(key);
- result = adapter.deleteObjects(keyList);
+ result = adapter.deleteObject(key);
}
if (result) {
diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketCommands.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketCommands.java
index ea4ec70..2d80060 100644
--- a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketCommands.java
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/BucketCommands.java
@@ -38,6 +38,7 @@ import picocli.CommandLine.ParentCommand;
InfoBucketHandler.class,
ListBucketHandler.class,
CreateBucketHandler.class,
+ LinkBucketHandler.class,
DeleteBucketHandler.class,
AddAclBucketHandler.class,
RemoveAclBucketHandler.class,
diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/LinkBucketHandler.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/LinkBucketHandler.java
new file mode 100644
index 0000000..6671f2d
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/LinkBucketHandler.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.bucket;
+
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.ozone.client.BucketArgs;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.shell.Handler;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Parameters;
+
+import java.io.IOException;
+
+/**
+ * Creates a symlink to another bucket.
+ */
+@Command(name = "link",
+ description = "creates a symlink to another bucket")
+public class LinkBucketHandler extends Handler {
+
+ @Parameters(index = "0", arity = "1..1",
+ description = "The bucket which the link should point to.",
+ converter = BucketUri.class)
+ private OzoneAddress source;
+
+ @Parameters(index = "1", arity = "1..1",
+ description = "Address of the link bucket",
+ converter = BucketUri.class)
+ private OzoneAddress target;
+
+ @Override
+ protected OzoneAddress getAddress() {
+ return source;
+ }
+
+ /**
+ * Executes create bucket.
+ */
+ @Override
+ public void execute(OzoneClient client, OzoneAddress address)
+ throws IOException {
+
+ BucketArgs.Builder bb = new BucketArgs.Builder()
+ .setStorageType(StorageType.DEFAULT)
+ .setVersioning(false)
+ .setSourceVolume(source.getVolumeName())
+ .setSourceBucket(source.getBucketName());
+
+ String volumeName = target.getVolumeName();
+ String bucketName = target.getBucketName();
+
+ OzoneVolume vol = client.getObjectStore().getVolume(volumeName);
+ vol.createBucket(bucketName, bb.build());
+
+ if (isVerbose()) {
+ OzoneBucket bucket = vol.getBucket(bucketName);
+ printObjectAsJson(bucket);
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 22/39: HDDS-3923. Display the safemode status on scm
page (#1165)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit c606a848905a78f115719a941dd3ef1997a44138
Author: maobaolong <30...@qq.com>
AuthorDate: Thu Jul 16 11:13:32 2020 +0800
HDDS-3923. Display the safemode status on scm page (#1165)
(cherry picked from commit 7266bf87da846241494cf3e219f1785f53f1c729)
---
.../java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java | 2 ++
.../hadoop/hdds/scm/server/StorageContainerManager.java | 10 ++++++++++
.../src/main/resources/webapps/scm/scm-overview.html | 11 +++++++++++
3 files changed, 23 insertions(+)
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java
index 2c63875..f0a497a 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java
@@ -64,4 +64,6 @@ public interface SCMMXBean extends ServiceRuntimeInfo {
* Returns the container count in all states.
*/
Map<String, Integer> getContainerStateCount();
+
+ Map<String, String> getRuleStatusMetrics();
}
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
index 0ed843c..621b126 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
@@ -1122,4 +1122,14 @@ public final class StorageContainerManager extends ServiceRuntimeInfoImpl
public Map<String, Pair<Boolean, String>> getRuleStatus() {
return scmSafeModeManager.getRuleStatus();
}
+
+ @Override
+ public Map<String, String> getRuleStatusMetrics() {
+ Map<String, String> map = new HashMap<>();
+ for (Map.Entry<String, Pair<Boolean, String>> entry :
+ scmSafeModeManager.getRuleStatus().entrySet()) {
+ map.put(entry.getKey(), entry.getValue().getRight());
+ }
+ return map;
+ }
}
diff --git a/hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html b/hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html
index 38ce638..ebb1170 100644
--- a/hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html
+++ b/hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html
@@ -42,3 +42,14 @@
</tr>
</tbody>
</table>
+
+<h2>Safemode rules statuses</h2>
+
+<table class="table table-bordered table-striped" class="col-md-6">
+ <tbody>
+ <tr ng-repeat="typestat in $ctrl.scmmetrics.RuleStatusMetrics">
+ <td>{{typestat.key}}</td>
+ <td>{{typestat.value}}</td>
+ </tr>
+ </tbody>
+</table>
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 39/39: HDDS-2770. security/SecurityAcls.md (#1190)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 5ac407364702ee5cd12010fa5f36783bdd391168
Author: Xiang Zhang <xi...@gmail.com>
AuthorDate: Wed Jul 22 20:48:39 2020 +0800
HDDS-2770. security/SecurityAcls.md (#1190)
(cherry picked from commit 05dccfd3800d3d08db4ed4b0e477d98bc7151012)
---
hadoop-hdds/docs/content/security/SecurityAcls.md | 8 +--
.../docs/content/security/SecurityAcls.zh.md | 66 ++++++++++++++++++++++
2 files changed, 70 insertions(+), 4 deletions(-)
diff --git a/hadoop-hdds/docs/content/security/SecurityAcls.md b/hadoop-hdds/docs/content/security/SecurityAcls.md
index 31bbb0a..b081a60 100644
--- a/hadoop-hdds/docs/content/security/SecurityAcls.md
+++ b/hadoop-hdds/docs/content/security/SecurityAcls.md
@@ -22,9 +22,9 @@ icon: transfer
limitations under the License.
-->
-Ozone supports a set of native ACLs. These ACLs can be used independently or
-along with Ranger. If Apache Ranger is enabled, then ACL will be checked
-first with Ranger and then Ozone's internal ACLs will be evaluated.
+Ozone supports a set of native ACLs. These ACLs can be used independently
+of ozone ACL plugin such as Ranger. If Apache Ranger plugin for Ozone is
+enabled, then ACL will be checked with Ranger.
Ozone ACLs are a super set of Posix and S3 ACLs.
@@ -52,7 +52,7 @@ we have no way of knowing who the user is or we don't care.
<div class="alert alert-success" role="alert">
- A S3 user accesing Ozone via AWS v4 signature protocol will be translated
+ A S3 user accessing Ozone via AWS v4 signature protocol will be translated
to the appropriate Kerberos user by Ozone Manager.
</div>
diff --git a/hadoop-hdds/docs/content/security/SecurityAcls.zh.md b/hadoop-hdds/docs/content/security/SecurityAcls.zh.md
new file mode 100644
index 0000000..9a3275b
--- /dev/null
+++ b/hadoop-hdds/docs/content/security/SecurityAcls.zh.md
@@ -0,0 +1,66 @@
+---
+title: "Ozone 访问控制列表"
+date: "2019-April-03"
+weight: 6
+summary: Ozone 原生的授权模块提供了不需要集成 Ranger 的访问控制列表（ACL）支持。
+icon: transfer
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Ozone 既支持原生的 ACL，也支持类似 Ranger 这样的 ACL 插件，如果启用了 Ranger 插件，则以 Ranger 中的 ACL 为准。
+
+Ozone 的 ACL 是 Posix ACL 和 S3 ACL 的超集。
+
+ACL 的通用格式为 _对象_:_角色_:_权限_.
+
+_对象_ 可选的值包括：
+
+1. **卷** - 一个 Ozone 卷，比如 _/volume_
+2. **桶** - 一个 Ozone 桶，比如 _/volume/bucket_
+3. **键** - 一个对象键，比如 _/volume/bucket/key_
+4. **前缀** - 某个键的路径前缀，比如 _/volume/bucket/prefix1/prefix2_
+
+_角色_ 可选的值包括:
+
+1. **用户** - 一个 Kerberos 用户，和 Posix 用户一样，用户可以是已创建的也可以是未创建的。
+2. **组** - 一个 Kerberos 组，和 Posix 组一样，组可以是已创建的也可以是未创建的。
+3. **所有人** - 所有通过 Kerberos 认证的用户，这对应 Posix 标准中的其它用户。
+4. **匿名** - 完全忽略用户字段，这是对 Posix 语义的扩展，使用 S3 协议时会用到，用于表达无法获取用户的身份或者不在乎用户的身份。
+
+<div class="alert alert-success" role="alert">
+ S3 用户通过 AWS v4 签名协议访问 Ozone 时，OM 会将其转化为对应的 Kerberos 用户。
+</div>
+
+_权限_ 可选的值包括：:
+
+1. **创建** – 此 ACL 为用户赋予在卷中创建桶，或者在桶中创建键的权限。请注意：在 Ozone 中，只有管理员可以创建卷。
+2. **列举** – 此 ACL 允许用户列举桶和键，因为列举的是子对象，所以这种 ACL 要绑定在卷和桶上。请注意：只有卷的属主和管理员可以对卷执行列举操作。
+3. **删除** – 允许用户删除卷、桶或键。
+4. **读取** – 允许用户读取卷和桶的元数据，以及读取键的数据流和元数据。
+5. **写入** - 允许用户修改卷和桶的元数据，以及重写一个已存在的键。
+6. **读 ACL** – 允许用户读取某个对象的 ACL。
+7. **写 ACL** – 允许用户修改某个对象的 ACL。
+
+<h3>Ozone 原生 ACL API</h3>
+
+ACL 可以通过 Ozone 提供的一系列 API 进行操作，支持的 API 包括：
+
+1. **SetAcl** – 此 API 的参数为用户主体、Ozone 对象名称、Ozone 对象的类型和 ACL 列表。
+2. **GetAcl** – 此 API 的参数为 Ozone 对象名称和 Ozone 对象类型，返回值为 ACL 列表。
+3. **AddAcl** - 此 API 的参数为 Ozone 对象名称、Ozone 对象类型和待添加的 ACL，新的 ACL 会被添加到该 Ozone 对象的 ACL 条目中。
+4. **RemoveAcl** - 此 API 的参数为 Ozone 对象名称、Ozone 对象类型和待删除的 ACL。
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 18/39: HDDS-3765. Fluentd writing to secure Ozone S3
API fails with 500 Error. (#1179)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit adb15ba882acd72c8540d30d29b0a576b9f8ca2f
Author: avijayanhwx <14...@users.noreply.github.com>
AuthorDate: Wed Jul 15 10:56:22 2020 -0700
HDDS-3765. Fluentd writing to secure Ozone S3 API fails with 500 Error. (#1179)
(cherry picked from commit d6c7f2836d2f2f06b2150299365311843eb58d08)
---
.../src/main/smoketest/security/ozone-secure-s3.robot | 15 +++++++++++++++
.../apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java | 8 +-------
2 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-s3.robot b/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-s3.robot
index 5103e80..70bade5 100644
--- a/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-s3.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-s3.robot
@@ -24,6 +24,8 @@ Test Timeout 5 minutes
*** Variables ***
${ENDPOINT_URL} http://s3g:9878
+${TEMPDIR} /tmp
+${TEST_FILE} NOTICE.txt
*** Keywords ***
Setup volume names
@@ -38,6 +40,19 @@ Secure S3 test Success
${output} = Execute aws s3api --endpoint-url ${ENDPOINT_URL} list-buckets
Should contain ${output} bucket-test123
+Secure S3 put-object test
+ ${testFilePath} = Set Variable ${TEMPDIR}/${TEST_FILE}
+ Copy File ${TEST_FILE} ${testFilePath}
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} put-object --bucket=bucket-test123 --key=tmp1/tmp2/NOTICE.txt --body=${testFilePath}
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} list-objects --bucket=bucket-test123
+ Should contain ${output} tmp1/tmp2/NOTICE.txt
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} put-object --bucket=bucket-test123 --key=tmp3//tmp4/NOTICE.txt --body=${testFilePath}
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} list-objects --bucket=bucket-test123
+ Should contain ${output} tmp3//tmp4/NOTICE.txt
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} put-object --bucket=bucket-test123 --key=//tmp5/tmp6/NOTICE.txt --body=${testFilePath}
+ ${output} = Execute aws s3api --endpoint ${ENDPOINT_URL} list-objects --bucket=bucket-test123
+ Should contain ${output} //tmp5/tmp6/NOTICE.txt
+
Secure S3 test Failure
Run Keyword Setup dummy credentials for S3
${rc} ${result} = Run And Return Rc And Output aws s3api --endpoint-url ${ENDPOINT_URL} create-bucket --bucket bucket-test123
diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java
index 099221d..1ff1a72 100644
--- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java
+++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java
@@ -104,13 +104,7 @@ public class AWSV4SignatureProcessor implements SignatureProcessor {
this.queryMap = context.getUriInfo().getQueryParameters();
- try {
- this.uri = new URI(context.getUriInfo().getRequestUri()
- .getPath().replaceAll("\\/+",
- "/")).normalize().getPath();
- } catch (URISyntaxException e) {
- throw S3_AUTHINFO_CREATION_ERROR;
- }
+ this.uri = context.getUriInfo().getRequestUri().getPath();
this.method = context.getMethod();
if (v4Header == null) {
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 28/39: HDDS-3965. SCM failed to start up for
duplicated pipeline detected. (#1210)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 6cdfc7d64755cfdf601d9da85493fecde5ed3e46
Author: avijayanhwx <14...@users.noreply.github.com>
AuthorDate: Fri Jul 17 11:51:44 2020 -0700
HDDS-3965. SCM failed to start up for duplicated pipeline detected. (#1210)
(cherry picked from commit ca4c5a154bfda6c176775138f809b17e6af1d77e)
---
.../hadoop/hdds/utils/db/RDBStoreIterator.java | 40 +++--
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 10 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 61 +++++++
.../hdds/scm/pipeline/SCMPipelineManager.java | 6 +
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 80 +++++++++
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 180 +++++++++++++++++++++
6 files changed, 360 insertions(+), 17 deletions(-)
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
index 5902486..ffe5f96 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStoreIterator.java
@@ -24,6 +24,8 @@ import java.util.NoSuchElementException;
import java.util.function.Consumer;
import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* RocksDB store iterator.
@@ -31,12 +33,16 @@ import org.rocksdb.RocksIterator;
public class RDBStoreIterator
implements TableIterator<byte[], ByteArrayKeyValue> {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RDBStoreIterator.class);
+
private RocksIterator rocksDBIterator;
private RDBTable rocksDBTable;
+ private ByteArrayKeyValue currentEntry;
public RDBStoreIterator(RocksIterator iterator) {
this.rocksDBIterator = iterator;
- rocksDBIterator.seekToFirst();
+ seekToFirst();
}
public RDBStoreIterator(RocksIterator iterator, RDBTable table) {
@@ -52,6 +58,15 @@ public class RDBStoreIterator
}
}
+ private void setCurrentEntry() {
+ if (rocksDBIterator.isValid()) {
+ currentEntry = ByteArrayKeyValue.create(rocksDBIterator.key(),
+ rocksDBIterator.value());
+ } else {
+ currentEntry = null;
+ }
+ }
+
@Override
public boolean hasNext() {
return rocksDBIterator.isValid();
@@ -59,12 +74,10 @@ public class RDBStoreIterator
@Override
public ByteArrayKeyValue next() {
- if (rocksDBIterator.isValid()) {
- ByteArrayKeyValue value =
- ByteArrayKeyValue.create(rocksDBIterator.key(), rocksDBIterator
- .value());
+ setCurrentEntry();
+ if (currentEntry != null) {
rocksDBIterator.next();
- return value;
+ return currentEntry;
}
throw new NoSuchElementException("RocksDB Store has no more elements");
}
@@ -72,21 +85,20 @@ public class RDBStoreIterator
@Override
public void seekToFirst() {
rocksDBIterator.seekToFirst();
+ setCurrentEntry();
}
@Override
public void seekToLast() {
rocksDBIterator.seekToLast();
+ setCurrentEntry();
}
@Override
public ByteArrayKeyValue seek(byte[] key) {
rocksDBIterator.seek(key);
- if (rocksDBIterator.isValid()) {
- return ByteArrayKeyValue.create(rocksDBIterator.key(),
- rocksDBIterator.value());
- }
- return null;
+ setCurrentEntry();
+ return currentEntry;
}
@Override
@@ -111,8 +123,10 @@ public class RDBStoreIterator
if (rocksDBTable == null) {
throw new UnsupportedOperationException("remove");
}
- if (rocksDBIterator.isValid()) {
- rocksDBTable.delete(rocksDBIterator.key());
+ if (currentEntry != null) {
+ rocksDBTable.delete(currentEntry.getKey());
+ } else {
+ LOG.info("Unable to delete currentEntry as it does not exist.");
}
}
diff --git a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
index 6e85977..fcb7dd2 100644
--- a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
+++ b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStoreIterator.java
@@ -58,12 +58,14 @@ public class TestRDBStoreIterator {
@Test
public void testForeachRemainingCallsConsumerWithAllElements() {
when(rocksDBIteratorMock.isValid())
- .thenReturn(true, true, true, true, true, true, false);
+ .thenReturn(true, true, true, true, true, true, true, false);
when(rocksDBIteratorMock.key())
- .thenReturn(new byte[]{0x00}, new byte[]{0x01}, new byte[]{0x02})
+ .thenReturn(new byte[]{0x00}, new byte[]{0x00}, new byte[]{0x01},
+ new byte[]{0x02})
.thenThrow(new NoSuchElementException());
when(rocksDBIteratorMock.value())
- .thenReturn(new byte[]{0x7f}, new byte[]{0x7e}, new byte[]{0x7d})
+ .thenReturn(new byte[]{0x7f}, new byte[]{0x7f}, new byte[]{0x7e},
+ new byte[]{0x7d})
.thenThrow(new NoSuchElementException());
@@ -91,7 +93,7 @@ public class TestRDBStoreIterator {
@Test
public void testHasNextDependsOnIsvalid(){
- when(rocksDBIteratorMock.isValid()).thenReturn(true, false);
+ when(rocksDBIteratorMock.isValid()).thenReturn(true, true, false);
RDBStoreIterator iter = new RDBStoreIterator(rocksDBIteratorMock);
diff --git a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java
index 00d05a1..5d00763 100644
--- a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java
+++ b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBTableStore.java
@@ -364,4 +364,65 @@ public class TestRDBTableStore {
Assert.assertTrue(keyCount > 0 && keyCount <= numKeys);
}
}
+
+ @Test
+ public void testIteratorRemoveFromDB() throws Exception {
+
+ // Remove without next removes first entry.
+ try (Table<byte[], byte[]> testTable = rdbStore.getTable("Fifth")) {
+ writeToTable(testTable, 3);
+ TableIterator<byte[], ? extends Table.KeyValue<byte[], byte[]>> iterator =
+ testTable.iterator();
+ iterator.removeFromDB();
+ Assert.assertNull(testTable.get("1".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNotNull(testTable.get("2".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNotNull(testTable.get("3".getBytes(StandardCharsets.UTF_8)));
+ }
+
+ // Remove after seekToLast removes lastEntry
+ try (Table<byte[], byte[]> testTable = rdbStore.getTable("Sixth")) {
+ writeToTable(testTable, 3);
+ TableIterator<byte[], ? extends Table.KeyValue<byte[], byte[]>> iterator =
+ testTable.iterator();
+ iterator.seekToLast();
+ iterator.removeFromDB();
+ Assert.assertNotNull(testTable.get("1".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNotNull(testTable.get("2".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNull(testTable.get("3".getBytes(StandardCharsets.UTF_8)));
+ }
+
+ // Remove after seek deletes that entry.
+ try (Table<byte[], byte[]> testTable = rdbStore.getTable("Sixth")) {
+ writeToTable(testTable, 3);
+ TableIterator<byte[], ? extends Table.KeyValue<byte[], byte[]>> iterator =
+ testTable.iterator();
+ iterator.seek("3".getBytes(StandardCharsets.UTF_8));
+ iterator.removeFromDB();
+ Assert.assertNotNull(testTable.get("1".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNotNull(testTable.get("2".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNull(testTable.get("3".getBytes(StandardCharsets.UTF_8)));
+ }
+
+ // Remove after next() deletes entry that was returned by next.
+ try (Table<byte[], byte[]> testTable = rdbStore.getTable("Sixth")) {
+ writeToTable(testTable, 3);
+ TableIterator<byte[], ? extends Table.KeyValue<byte[], byte[]>> iterator =
+ testTable.iterator();
+ iterator.seek("2".getBytes(StandardCharsets.UTF_8));
+ iterator.next();
+ iterator.removeFromDB();
+ Assert.assertNotNull(testTable.get("1".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNull(testTable.get("2".getBytes(StandardCharsets.UTF_8)));
+ Assert.assertNotNull(testTable.get("3".getBytes(StandardCharsets.UTF_8)));
+ }
+ }
+
+ private void writeToTable(Table testTable, int num) throws IOException {
+ for (int i = 1; i <= num; i++) {
+ byte[] key = (i + "").getBytes(StandardCharsets.UTF_8);
+ byte[] value =
+ RandomStringUtils.random(10).getBytes(StandardCharsets.UTF_8);
+ testTable.put(key, value);
+ }
+ }
}
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
index fda9371..6fce895 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
@@ -209,6 +209,7 @@ public class SCMPipelineManager implements PipelineManager {
) {
if (!pipelineID.equals(pipeline.getId())) {
try {
+ LOG.info("Found pipeline in old format key : {}", pipeline.getId());
it.removeFromDB();
pipelineStore.put(pipeline.getId(), pipeline);
} catch (IOException e) {
@@ -701,4 +702,9 @@ public class SCMPipelineManager implements PipelineManager {
startPipelineCreator();
}
}
+
+ @VisibleForTesting
+ protected static Logger getLog() {
+ return LOG;
+ }
}
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
index fc8f61a..62289b9 100644
--- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
@@ -22,10 +22,13 @@ import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Set;
+import java.util.UUID;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.stream.Collectors;
@@ -56,12 +59,15 @@ import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_DATANODE_PIPELINE_L
import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_PIPELINE_ALLOCATED_TIMEOUT;
import static org.apache.hadoop.test.MetricsAsserts.getLongCounter;
import static org.apache.hadoop.test.MetricsAsserts.getMetrics;
+
+import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
import org.junit.After;
import org.junit.Assert;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
import org.junit.Before;
import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
import org.mockito.InOrder;
import static org.mockito.Mockito.doReturn;
@@ -617,6 +623,80 @@ public class TestSCMPipelineManager {
verify(pipelineStore, never()).put(p2.getId(), p2);
}
+ @Test
+ public void testScmWithPipelineDBKeyFormatChange() throws Exception {
+ TemporaryFolder tempDir = new TemporaryFolder();
+ tempDir.create();
+ File dir = tempDir.newFolder();
+ conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, dir.getAbsolutePath());
+
+ SCMMetadataStore scmDbWithOldKeyFormat = null;
+ Map<UUID, Pipeline> oldPipelines = new HashMap<>();
+ try {
+ scmDbWithOldKeyFormat =
+ new TestSCMStoreImplWithOldPipelineIDKeyFormat(conf);
+ // Create 3 pipelines.
+ for (int i = 0; i < 3; i++) {
+ Pipeline pipeline = pipelineStub();
+ scmDbWithOldKeyFormat.getPipelineTable()
+ .put(pipeline.getId(), pipeline);
+ oldPipelines.put(pipeline.getId().getId(), pipeline);
+ }
+ } finally {
+ if (scmDbWithOldKeyFormat != null) {
+ scmDbWithOldKeyFormat.stop();
+ }
+ }
+
+ LogCapturer logCapturer =
+ LogCapturer.captureLogs(SCMPipelineManager.getLog());
+
+ // Create SCMPipelineManager with new DBDefinition.
+ SCMMetadataStore newScmMetadataStore = null;
+ try {
+ newScmMetadataStore = new SCMMetadataStoreImpl(conf);
+ SCMPipelineManager pipelineManager = new SCMPipelineManager(conf,
+ nodeManager,
+ newScmMetadataStore.getPipelineTable(),
+ new EventQueue());
+
+ waitForLog(logCapturer);
+ assertEquals(3, pipelineManager.getPipelines().size());
+ oldPipelines.values().forEach(p ->
+ pipelineManager.containsPipeline(p.getId()));
+ } finally {
+ newScmMetadataStore.stop();
+ }
+
+ // Mimicking another restart.
+ try {
+ logCapturer.clearOutput();
+ newScmMetadataStore = new SCMMetadataStoreImpl(conf);
+ SCMPipelineManager pipelineManager = new SCMPipelineManager(conf,
+ nodeManager,
+ newScmMetadataStore.getPipelineTable(),
+ new EventQueue());
+ try {
+ waitForLog(logCapturer);
+ Assert.fail("Unexpected log: " + logCapturer.getOutput());
+ } catch (TimeoutException ex) {
+ Assert.assertTrue(ex.getMessage().contains("Timed out"));
+ }
+ assertEquals(3, pipelineManager.getPipelines().size());
+ oldPipelines.values().forEach(p ->
+ pipelineManager.containsPipeline(p.getId()));
+ } finally {
+ newScmMetadataStore.stop();
+ }
+ }
+
+ private static void waitForLog(LogCapturer logCapturer)
+ throws TimeoutException, InterruptedException {
+ GenericTestUtils.waitFor(() -> logCapturer.getOutput()
+ .contains("Found pipeline in old format key"),
+ 1000, 5000);
+ }
+
private Pipeline pipelineStub() {
return Pipeline.newBuilder()
.setId(PipelineID.randomId())
diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java
new file mode 100644
index 0000000..a04ecea
--- /dev/null
+++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java
@@ -0,0 +1,180 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.pipeline;
+
+import static org.apache.hadoop.hdds.scm.metadata.SCMDBDefinition.PIPELINES;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.security.cert.X509Certificate;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.DeletedBlocksTransaction;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.metadata.PipelineCodec;
+import org.apache.hadoop.hdds.scm.metadata.SCMMetadataStore;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateStore;
+import org.apache.hadoop.hdds.utils.db.BatchOperationHandler;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+import org.apache.hadoop.hdds.utils.db.DBStoreBuilder;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+/**
+ * Test SCM Metadata Store that has ONLY the pipeline table whose key uses the
+ * old codec format.
+ */
+public class TestSCMStoreImplWithOldPipelineIDKeyFormat
+ implements SCMMetadataStore {
+
+ private DBStore store;
+ private final OzoneConfiguration configuration;
+ private Table<PipelineID, Pipeline> pipelineTable;
+
+ public TestSCMStoreImplWithOldPipelineIDKeyFormat(
+ OzoneConfiguration config) throws IOException {
+ this.configuration = config;
+ start(configuration);
+ }
+
+ @Override
+ public void start(OzoneConfiguration config)
+ throws IOException {
+ if (this.store == null) {
+ this.store = DBStoreBuilder.createDBStore(config,
+ new SCMDBTestDefinition());
+ pipelineTable = PIPELINES.getTable(store);
+ }
+ }
+
+ @Override
+ public void stop() throws Exception {
+ if (store != null) {
+ store.close();
+ store = null;
+ }
+ }
+
+ @Override
+ public DBStore getStore() {
+ return null;
+ }
+
+ @Override
+ public Table<Long, DeletedBlocksTransaction> getDeletedBlocksTXTable() {
+ return null;
+ }
+
+ @Override
+ public Long getCurrentTXID() {
+ return null;
+ }
+
+ @Override
+ public Long getNextDeleteBlockTXID() {
+ return null;
+ }
+
+ @Override
+ public Table<BigInteger, X509Certificate> getValidCertsTable() {
+ return null;
+ }
+
+ @Override
+ public Table<BigInteger, X509Certificate> getRevokedCertsTable() {
+ return null;
+ }
+
+ @Override
+ public TableIterator getAllCerts(CertificateStore.CertType certType) {
+ return null;
+ }
+
+ @Override
+ public Table<PipelineID, Pipeline> getPipelineTable() {
+ return pipelineTable;
+ }
+
+ @Override
+ public BatchOperationHandler getBatchHandler() {
+ return null;
+ }
+
+ @Override
+ public Table<ContainerID, ContainerInfo> getContainerTable() {
+ return null;
+ }
+
+ /**
+ * Test SCM DB Definition for the above class.
+ */
+ public static class SCMDBTestDefinition implements DBDefinition {
+
+ public static final DBColumnFamilyDefinition<PipelineID, Pipeline>
+ PIPELINES =
+ new DBColumnFamilyDefinition<>(
+ "pipelines",
+ PipelineID.class,
+ new OldPipelineIDCodec(),
+ Pipeline.class,
+ new PipelineCodec());
+
+ @Override
+ public String getName() {
+ return "scm.db";
+ }
+
+ @Override
+ public String getLocationConfigKey() {
+ return ScmConfigKeys.OZONE_SCM_DB_DIRS;
+ }
+
+ @Override
+ public DBColumnFamilyDefinition[] getColumnFamilies() {
+ return new DBColumnFamilyDefinition[] {PIPELINES};
+ }
+ }
+
+ /**
+ * Old Pipeline ID codec that relies on protobuf serialization.
+ */
+ public static class OldPipelineIDCodec implements Codec<PipelineID> {
+ @Override
+ public byte[] toPersistedFormat(PipelineID object) throws IOException {
+ return object.getProtobuf().toByteArray();
+ }
+
+ @Override
+ public PipelineID fromPersistedFormat(byte[] rawData) throws IOException {
+ return null;
+ }
+
+ @Override
+ public PipelineID copyObject(PipelineID object) {
+ throw new UnsupportedOperationException();
+ }
+ }
+
+}
+
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 01/39: HDDS-3914. Remove LevelDB configuration
option for DN Metastore (#1166)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit c9e5a7f806c52d56d660a65ba9de6fc22936b961
Author: Hanisha Koneru <ha...@apache.org>
AuthorDate: Fri Jul 10 12:50:15 2020 -0700
HDDS-3914. Remove LevelDB configuration option for DN Metastore (#1166)
(cherry picked from commit 2af6198686d81daa3ad0513f723118637d2945cf)
---
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 9 ----
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../common/src/main/resources/ozone-default.xml | 20 ---------
.../container/keyvalue/KeyValueContainer.java | 5 ---
.../container/keyvalue/KeyValueContainerCheck.java | 8 ++--
.../container/keyvalue/KeyValueContainerData.java | 3 +-
.../keyvalue/TestKeyValueBlockIterator.java | 16 ++------
.../keyvalue/TestKeyValueContainerCheck.java | 19 ++-------
.../org/apache/hadoop/hdds/utils/LevelDBStore.java | 1 +
.../hadoop/hdds/utils/LevelDBStoreIterator.java | 1 +
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 25 +++++------
.../hadoop/hdds/utils/TestMetadataStore.java | 48 ++++++++++++----------
.../hadoop/hdds/utils/TestRocksDBStoreMBean.java | 3 --
.../org/apache/hadoop/ozone/om/TestOmSQLCli.java | 21 ----------
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 3 --
.../hadoop/ozone/recon/ReconServerConfigKeys.java | 7 ----
16 files changed, 55 insertions(+), 136 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index 15dd65f..dc8b231 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -79,15 +79,6 @@ public final class OzoneConfigKeys {
"ozone.trace.enabled";
public static final boolean OZONE_TRACE_ENABLED_DEFAULT = false;
- public static final String OZONE_METADATA_STORE_IMPL =
- "ozone.metastore.impl";
- public static final String OZONE_METADATA_STORE_IMPL_LEVELDB =
- "LevelDB";
- public static final String OZONE_METADATA_STORE_IMPL_ROCKSDB =
- "RocksDB";
- public static final String OZONE_METADATA_STORE_IMPL_DEFAULT =
- OZONE_METADATA_STORE_IMPL_ROCKSDB;
-
public static final String OZONE_METADATA_STORE_ROCKSDB_STATISTICS =
"ozone.metastore.rocksdb.statistics";
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
index 34bcd3f..a473948 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
@@ -362,4 +362,6 @@ public final class OzoneConsts {
public static final String TRANSACTION_INFO_KEY = "#TRANSACTIONINFO";
public static final String TRANSACTION_INFO_SPLIT_KEY = "#";
+ public static final String CONTAINER_DB_TYPE_ROCKSDB = "RocksDB";
+ public static final String CONTAINER_DB_TYPE_LEVELDB = "LevelDB";
}
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index 86e8bfb..d8fc591 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -634,17 +634,6 @@
dfs.container.ratis.datanode.storage.dir be configured separately.
</description>
</property>
- <property>
- <name>ozone.metastore.impl</name>
- <value>RocksDB</value>
- <tag>OZONE, OM, SCM, CONTAINER, STORAGE</tag>
- <description>
- Ozone metadata store implementation. Ozone metadata are well
- distributed to multiple services such as ozoneManager, scm. They are stored in
- some local key-value databases. This property determines which database
- library to use. Supported value is either LevelDB or RocksDB.
- </description>
- </property>
<property>
<name>ozone.metastore.rocksdb.statistics</name>
@@ -2292,15 +2281,6 @@
</description>
</property>
<property>
- <name>ozone.recon.container.db.impl</name>
- <value>RocksDB</value>
- <tag>OZONE, RECON, STORAGE</tag>
- <description>
- Ozone Recon container DB store implementation.Supported value is either
- LevelDB or RocksDB.
- </description>
- </property>
- <property>
<name>ozone.recon.om.db.dir</name>
<value/>
<tag>OZONE, RECON, STORAGE</tag>
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
index 49b907f..a80841f 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
@@ -39,7 +39,6 @@ import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerExcep
import org.apache.hadoop.hdfs.util.Canceler;
import org.apache.hadoop.hdfs.util.DataTransferThrottler;
import org.apache.hadoop.io.nativeio.NativeIO;
-import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.OzoneConsts;
import org.apache.hadoop.ozone.container.common.helpers.ContainerUtils;
import org.apache.hadoop.ozone.container.common.impl.ContainerDataYaml;
@@ -129,12 +128,8 @@ public class KeyValueContainer implements Container<KeyValueContainerData> {
KeyValueContainerUtil.createContainerMetaData(containerMetaDataPath,
chunksPath, dbFile, config);
- String impl = config.getTrimmed(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL,
- OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_DEFAULT);
-
//Set containerData for the KeyValueContainer.
containerData.setChunksPath(chunksPath.getPath());
- containerData.setContainerDBType(impl);
containerData.setDbFile(dbFile);
containerData.setVolume(containerVolume);
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java
index 95795e6..d6c4ff0 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java
@@ -47,8 +47,8 @@ import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_LEVELDB;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_ROCKSDB;
/**
* Class to run integrity checks on Datanode Containers.
@@ -186,8 +186,8 @@ public class KeyValueContainerCheck {
}
dbType = onDiskContainerData.getContainerDBType();
- if (!dbType.equals(OZONE_METADATA_STORE_IMPL_ROCKSDB) &&
- !dbType.equals(OZONE_METADATA_STORE_IMPL_LEVELDB)) {
+ if (!dbType.equals(CONTAINER_DB_TYPE_ROCKSDB) &&
+ !dbType.equals(CONTAINER_DB_TYPE_LEVELDB)) {
String errStr = "Unknown DBType [" + dbType
+ "] in Container File for [" + containerID + "]";
throw new IOException(errStr);
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
index 373b322..5698d72 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
@@ -41,6 +41,7 @@ import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import static java.lang.Math.max;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_ROCKSDB;
import static org.apache.hadoop.ozone.OzoneConsts.DB_BLOCK_COUNT_KEY;
import static org.apache.hadoop.ozone.OzoneConsts.CHUNKS_PATH;
import static org.apache.hadoop.ozone.OzoneConsts.DB_CONTAINER_BYTES_USED_KEY;
@@ -65,7 +66,7 @@ public class KeyValueContainerData extends ContainerData {
private String metadataPath;
//Type of DB used to store key to chunks mapping
- private String containerDBType;
+ private String containerDBType = CONTAINER_DB_TYPE_ROCKSDB;
private File dbFile = null;
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java
index 62097b3..aff0528 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueBlockIterator.java
@@ -45,9 +45,6 @@ import org.apache.hadoop.test.GenericTestUtils;
import com.google.common.primitives.Longs;
import static org.apache.hadoop.hdds.scm.ScmConfigKeys.HDDS_DATANODE_DIR_KEY;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB;
import static org.apache.hadoop.ozone.container.common.impl.ChunkLayOutVersion.FILE_PER_BLOCK;
import static org.apache.hadoop.ozone.container.common.impl.ChunkLayOutVersion.FILE_PER_CHUNK;
import org.junit.After;
@@ -71,22 +68,18 @@ public class TestKeyValueBlockIterator {
private OzoneConfiguration conf;
private File testRoot;
- private final String storeImpl;
private final ChunkLayOutVersion layout;
- public TestKeyValueBlockIterator(String metadataImpl,
- ChunkLayOutVersion layout) {
- this.storeImpl = metadataImpl;
+ public TestKeyValueBlockIterator(ChunkLayOutVersion layout) {
this.layout = layout;
}
@Parameterized.Parameters
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][] {
- {OZONE_METADATA_STORE_IMPL_LEVELDB, FILE_PER_CHUNK},
- {OZONE_METADATA_STORE_IMPL_ROCKSDB, FILE_PER_CHUNK},
- {OZONE_METADATA_STORE_IMPL_LEVELDB, FILE_PER_BLOCK},
- {OZONE_METADATA_STORE_IMPL_ROCKSDB, FILE_PER_BLOCK}});
+ {FILE_PER_CHUNK},
+ {FILE_PER_BLOCK}
+ });
}
@Before
@@ -94,7 +87,6 @@ public class TestKeyValueBlockIterator {
testRoot = GenericTestUtils.getRandomizedTestDir();
conf = new OzoneConfiguration();
conf.set(HDDS_DATANODE_DIR_KEY, testRoot.getAbsolutePath());
- conf.set(OZONE_METADATA_STORE_IMPL, storeImpl);
volumeSet = new MutableVolumeSet(UUID.randomUUID().toString(), conf);
}
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
index cb8ef34..4583a54 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
@@ -58,10 +58,6 @@ import java.util.List;
import java.util.UUID;
import static org.apache.hadoop.hdds.scm.ScmConfigKeys.HDDS_DATANODE_DIR_KEY;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL;
-
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
@@ -76,7 +72,6 @@ import static org.junit.Assert.assertFalse;
private static final Logger LOG =
LoggerFactory.getLogger(TestKeyValueContainerCheck.class);
- private final String storeImpl;
private final ChunkLayoutTestInfo chunkManagerTestInfo;
private KeyValueContainer container;
private KeyValueContainerData containerData;
@@ -85,28 +80,22 @@ import static org.junit.Assert.assertFalse;
private File testRoot;
private ChunkManager chunkManager;
- public TestKeyValueContainerCheck(String metadataImpl,
- ChunkLayoutTestInfo chunkManagerTestInfo) {
- this.storeImpl = metadataImpl;
+ public TestKeyValueContainerCheck(ChunkLayoutTestInfo chunkManagerTestInfo) {
this.chunkManagerTestInfo = chunkManagerTestInfo;
}
@Parameterized.Parameters public static Collection<Object[]> data() {
return Arrays.asList(new Object[][] {
- {OZONE_METADATA_STORE_IMPL_LEVELDB, ChunkLayoutTestInfo.FILE_PER_CHUNK},
- {OZONE_METADATA_STORE_IMPL_LEVELDB, ChunkLayoutTestInfo.FILE_PER_BLOCK},
- {OZONE_METADATA_STORE_IMPL_ROCKSDB, ChunkLayoutTestInfo.FILE_PER_CHUNK},
- {OZONE_METADATA_STORE_IMPL_ROCKSDB, ChunkLayoutTestInfo.FILE_PER_BLOCK}
+ {ChunkLayoutTestInfo.FILE_PER_CHUNK},
+ {ChunkLayoutTestInfo.FILE_PER_BLOCK}
});
}
@Before public void setUp() throws Exception {
- LOG.info("Testing store:{} layout:{}",
- storeImpl, chunkManagerTestInfo.getLayout());
+ LOG.info("Testing layout:{}", chunkManagerTestInfo.getLayout());
this.testRoot = GenericTestUtils.getRandomizedTestDir();
conf = new OzoneConfiguration();
conf.set(HDDS_DATANODE_DIR_KEY, testRoot.getAbsolutePath());
- conf.set(OZONE_METADATA_STORE_IMPL, storeImpl);
chunkManagerTestInfo.updateConfig(conf);
volumeSet = new MutableVolumeSet(UUID.randomUUID().toString(), conf);
chunkManager = chunkManagerTestInfo.createChunkManager(true, null);
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java
index f508868..8998f6c 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java
@@ -42,6 +42,7 @@ import java.util.Map.Entry;
/**
* LevelDB interface.
*/
+@Deprecated
public class LevelDBStore implements MetadataStore {
private static final Logger LOG =
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java
index f5b6769..0a7abe3 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java
@@ -25,6 +25,7 @@ import java.util.NoSuchElementException;
/**
* LevelDB store iterator.
*/
+@Deprecated
public class LevelDBStoreIterator
implements MetaStoreIterator<MetadataStore.KeyValue> {
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java
index d697fdf..5e1a5a8 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java
@@ -26,14 +26,14 @@ import java.util.concurrent.ConcurrentHashMap;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
-import org.apache.hadoop.ozone.OzoneConfigKeys;
import com.google.common.annotations.VisibleForTesting;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_DEFAULT;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_LEVELDB;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_ROCKSDB;
+
import org.iq80.leveldb.Options;
import org.rocksdb.BlockBasedTableConfig;
import org.rocksdb.Statistics;
@@ -104,22 +104,19 @@ public class MetadataStoreBuilder {
final ConfigurationSource conf = optionalConf.orElse(DEFAULT_CONF);
if (dbType == null) {
- LOG.debug("dbType is null, using ");
- dbType = conf.getTrimmed(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL,
- OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_DEFAULT);
- LOG.debug("dbType is null, using dbType {} from ozone configuration",
- dbType);
+ dbType = CONTAINER_DB_TYPE_ROCKSDB;
+ LOG.debug("dbType is null, using dbType {}.", dbType);
} else {
LOG.debug("Using dbType {} for metastore", dbType);
}
- if (OZONE_METADATA_STORE_IMPL_LEVELDB.equals(dbType)) {
+ if (CONTAINER_DB_TYPE_LEVELDB.equals(dbType)) {
Options options = new Options();
options.createIfMissing(createIfMissing);
if (cacheSize > 0) {
options.cacheSize(cacheSize);
}
return new LevelDBStore(dbFile, options);
- } else if (OZONE_METADATA_STORE_IMPL_ROCKSDB.equals(dbType)) {
+ } else if (CONTAINER_DB_TYPE_ROCKSDB.equals(dbType)) {
org.rocksdb.Options opts;
// Used cached options if config object passed down is the same
if (CACHED_OPTS.containsKey(conf)) {
@@ -147,10 +144,8 @@ public class MetadataStoreBuilder {
return new RocksDBStore(dbFile, opts);
}
- throw new IllegalArgumentException("Invalid argument for "
- + OzoneConfigKeys.OZONE_METADATA_STORE_IMPL
- + ". Expecting " + OZONE_METADATA_STORE_IMPL_LEVELDB
- + " or " + OZONE_METADATA_STORE_IMPL_ROCKSDB
- + ", but met " + dbType);
+ throw new IllegalArgumentException("Invalid Container DB type. Expecting "
+ + CONTAINER_DB_TYPE_LEVELDB + " or "
+ + CONTAINER_DB_TYPE_ROCKSDB + ", but met " + dbType);
}
}
diff --git a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java
index 3eb832f..ed9bfb3 100644
--- a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java
+++ b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java
@@ -32,28 +32,30 @@ import org.apache.hadoop.hdds.StringUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.utils.MetadataKeyFilters.KeyPrefixFilter;
import org.apache.hadoop.hdds.utils.MetadataKeyFilters.MetadataKeyFilter;
-import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.test.GenericTestUtils;
import com.google.common.collect.Lists;
-import static java.nio.charset.StandardCharsets.UTF_8;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.tuple.ImmutablePair;
-import static org.apache.hadoop.test.PlatformAssumptions.assumeNotWindows;
+
import org.junit.After;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
-import static org.junit.runners.Parameterized.Parameters;
import org.slf4j.event.Level;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_LEVELDB;
+import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_ROCKSDB;
+import static org.apache.hadoop.test.PlatformAssumptions.assumeNotWindows;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.junit.runners.Parameterized.Parameters;
+import static java.nio.charset.StandardCharsets.UTF_8;
/**
* Test class for ozone metadata store.
*/
@@ -74,14 +76,14 @@ public class TestMetadataStore {
@Parameters
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][] {
- {OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB},
- {OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB}
+ {CONTAINER_DB_TYPE_LEVELDB},
+ {CONTAINER_DB_TYPE_ROCKSDB}
});
}
@Before
public void init() throws IOException {
- if (OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB.equals(storeImpl)) {
+ if (CONTAINER_DB_TYPE_ROCKSDB.equals(storeImpl)) {
// The initialization of RocksDB fails on Windows
assumeNotWindows();
}
@@ -90,12 +92,12 @@ public class TestMetadataStore {
+ "-" + storeImpl.toLowerCase());
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
store = MetadataStoreBuilder.newBuilder()
.setConf(conf)
.setCreateIfMissing(true)
.setDbFile(testDir)
+ .setDBType(storeImpl)
.build();
// Add 20 entries.
@@ -110,12 +112,13 @@ public class TestMetadataStore {
@Test
public void testIterator() throws Exception {
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
+
File dbDir = GenericTestUtils.getRandomizedTestDir();
MetadataStore dbStore = MetadataStoreBuilder.newBuilder()
.setConf(conf)
.setCreateIfMissing(true)
.setDbFile(dbDir)
+ .setDBType(storeImpl)
.build();
//As database is empty, check whether iterator is working as expected or
@@ -166,15 +169,15 @@ public class TestMetadataStore {
public void testMetaStoreConfigDifferentFromType() throws IOException {
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
+
String dbType;
GenericTestUtils.setLogLevel(MetadataStoreBuilder.LOG, Level.DEBUG);
GenericTestUtils.LogCapturer logCapturer =
GenericTestUtils.LogCapturer.captureLogs(MetadataStoreBuilder.LOG);
- if (storeImpl.equals(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB)) {
- dbType = "RocksDB";
- } else {
+ if (storeImpl.equals(CONTAINER_DB_TYPE_LEVELDB)) {
dbType = "LevelDB";
+ } else {
+ dbType = "RocksDB";
}
File dbDir = GenericTestUtils.getTestDir(getClass().getSimpleName()
@@ -193,7 +196,7 @@ public class TestMetadataStore {
public void testdbTypeNotSet() throws IOException {
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
+
GenericTestUtils.setLogLevel(MetadataStoreBuilder.LOG, Level.DEBUG);
GenericTestUtils.LogCapturer logCapturer =
GenericTestUtils.LogCapturer.captureLogs(MetadataStoreBuilder.LOG);
@@ -203,7 +206,7 @@ public class TestMetadataStore {
MetadataStore dbStore = MetadataStoreBuilder.newBuilder().setConf(conf)
.setCreateIfMissing(true).setDbFile(dbDir).build();
assertTrue(logCapturer.getOutput().contains("dbType is null, using dbType" +
- " " + storeImpl));
+ " " + CONTAINER_DB_TYPE_ROCKSDB));
dbStore.close();
dbStore.destroy();
FileUtils.deleteDirectory(dbDir);
@@ -213,8 +216,11 @@ public class TestMetadataStore {
@After
public void cleanup() throws IOException {
if (store != null) {
+ System.out.println("--- Closing Store: " + store.getClass());
store.close();
store.destroy();
+ } else {
+ System.out.println("--- Store already closed: " + store.getClass());
}
if (testDir != null) {
FileUtils.deleteDirectory(testDir);
@@ -460,7 +466,6 @@ public class TestMetadataStore {
public void testDestroyDB() throws IOException {
// create a new DB to test db destroy
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
File dbDir = GenericTestUtils.getTestDir(getClass().getSimpleName()
+ "-" + storeImpl.toLowerCase() + "-toDestroy");
@@ -468,6 +473,7 @@ public class TestMetadataStore {
.setConf(conf)
.setCreateIfMissing(true)
.setDbFile(dbDir)
+ .setDBType(storeImpl)
.build();
dbStore.put(getBytes("key1"), getBytes("value1"));
@@ -485,7 +491,6 @@ public class TestMetadataStore {
@Test
public void testBatchWrite() throws IOException {
OzoneConfiguration conf = new OzoneConfiguration();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, storeImpl);
File dbDir = GenericTestUtils.getTestDir(getClass().getSimpleName()
+ "-" + storeImpl.toLowerCase() + "-batchWrite");
@@ -493,6 +498,7 @@ public class TestMetadataStore {
.setConf(conf)
.setCreateIfMissing(true)
.setDbFile(dbDir)
+ .setDBType(storeImpl)
.build();
List<String> expectedResult = Lists.newArrayList();
diff --git a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java
index 610e898..8b3554a 100644
--- a/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java
+++ b/hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java
@@ -53,9 +53,6 @@ public class TestRocksDBStoreMBean {
@Before
public void init() throws Exception {
conf = new OzoneConfiguration();
-
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL,
- OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB);
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java
index a082e99..b1ce4ba 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java
@@ -26,7 +26,6 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.UUID;
@@ -34,7 +33,6 @@ import java.util.UUID;
import org.apache.hadoop.hdds.HddsConfigKeys;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.MiniOzoneCluster;
-import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.TestDataUtil;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.scm.cli.SQLCLI;
@@ -51,13 +49,10 @@ import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
/**
* This class tests the CLI that transforms om.db into SQLite DB files.
*/
-@RunWith(Parameterized.class)
public class TestOmSQLCli {
/**
@@ -82,21 +77,6 @@ public class TestOmSQLCli {
private String keyName2 = "key2";
private String keyName3 = "key3";
- @Parameterized.Parameters
- public static Collection<Object[]> data() {
- return Arrays.asList(new Object[][] {
- // Uncomment the below line if we support leveldb in future.
- //{OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB},
- {OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB}
- });
- }
-
- private String metaStoreType;
-
- public TestOmSQLCli(String type) {
- metaStoreType = type;
- }
-
/**
* Create a MiniDFSCluster for testing.
* <p>
@@ -123,7 +103,6 @@ public class TestOmSQLCli {
cluster.getOzoneManager().stop();
cluster.getStorageContainerManager().stop();
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL, metaStoreType);
cli = new SQLCLI(conf);
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java
index 6c858ab..d3e228f 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java
@@ -35,7 +35,6 @@ import org.apache.hadoop.hdds.scm.server.StorageContainerManager;
import org.apache.hadoop.hdds.server.events.EventQueue;
import org.apache.hadoop.ozone.HddsDatanodeService;
import org.apache.hadoop.ozone.MiniOzoneCluster;
-import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.TestStorageContainerManagerHelper;
import org.apache.hadoop.ozone.client.ObjectStore;
import org.apache.hadoop.ozone.client.OzoneBucket;
@@ -289,8 +288,6 @@ public class TestScmSafeMode {
@Test(timeout = 300_000)
public void testSCMSafeModeRestrictedOp() throws Exception {
- conf.set(OzoneConfigKeys.OZONE_METADATA_STORE_IMPL,
- OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_LEVELDB);
cluster.stop();
cluster = builder.build();
StorageContainerManager scm = cluster.getStorageContainerManager();
diff --git a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java
index 3605896..704c18e 100644
--- a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java
+++ b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java
@@ -17,8 +17,6 @@
*/
package org.apache.hadoop.ozone.recon;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_IMPL_ROCKSDB;
-
import org.apache.hadoop.hdds.annotation.InterfaceAudience;
import org.apache.hadoop.hdds.annotation.InterfaceStability;
@@ -80,11 +78,6 @@ public final class ReconServerConfigKeys {
public static final String
RECON_OM_SNAPSHOT_TASK_INITIAL_DELAY_DEFAULT = "1m";
- public static final String OZONE_RECON_CONTAINER_DB_STORE_IMPL =
- "ozone.recon.container.db.impl";
- public static final String OZONE_RECON_CONTAINER_DB_STORE_IMPL_DEFAULT =
- OZONE_METADATA_STORE_IMPL_ROCKSDB;
-
public static final String RECON_OM_SNAPSHOT_TASK_INTERVAL =
"recon.om.snapshot.task.interval.delay";
public static final String RECON_OM_SNAPSHOT_TASK_INTERVAL_DEFAULT
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 19/39: HDDS-3967. Remove leftover debug setting
(#1202)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit e65d196b40fa11e0d601743e9aa66c077ef5da55
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Thu Jul 16 01:26:15 2020 +0200
HDDS-3967. Remove leftover debug setting (#1202)
(cherry picked from commit fadc7f1195a459d67ed25a3ea6829fede9b029dc)
---
hadoop-ozone/integration-test/src/test/resources/log4j.properties | 3 ---
1 file changed, 3 deletions(-)
diff --git a/hadoop-ozone/integration-test/src/test/resources/log4j.properties b/hadoop-ozone/integration-test/src/test/resources/log4j.properties
index e5381ad..b8ad21d 100644
--- a/hadoop-ozone/integration-test/src/test/resources/log4j.properties
+++ b/hadoop-ozone/integration-test/src/test/resources/log4j.properties
@@ -19,6 +19,3 @@ log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} (%F:%
log4j.logger.org.apache.hadoop.security.ShellBasedUnixGroupsMapping=ERROR
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
-
-log4j.logger.org.apache.ratis.grpc.server.GrpcLogAppender=DEBUG
-log4j.logger.org.apache.ratis.server.impl.RaftServerImpl=DEBUG
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 37/39: HDDS-3892. Datanode initialization is too slow when there are thousan… (#1147)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 2a79c8977d85c6c6e86d840325250e09c3be0b27
Author: Sammi Chen <sa...@apache.org>
AuthorDate: Wed Jul 22 14:11:08 2020 +0800
HDDS-3892. Datanode initialization is too slow when there are thousan… (#1147)
(cherry picked from commit 783a18c4a47c2b1eccbc85611211753943598ea1)
---
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 3 +
.../common/src/main/resources/ozone-default.xml | 10 +++
.../container/common/utils/ContainerCache.java | 71 ++++++++++++++++------
.../ozone/container/ozoneimpl/ContainerReader.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 3 +
.../ozone/container/common/TestContainerCache.java | 52 ++++++++++++++++
.../container/ozoneimpl/TestContainerReader.java | 69 ++++++++++++++++++++-
7 files changed, 189 insertions(+), 21 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index dc8b231..d89fef9 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -95,6 +95,9 @@ public final class OzoneConfigKeys {
public static final String OZONE_CONTAINER_CACHE_SIZE =
"ozone.container.cache.size";
public static final int OZONE_CONTAINER_CACHE_DEFAULT = 1024;
+ public static final String OZONE_CONTAINER_CACHE_LOCK_STRIPES =
+ "ozone.container.cache.lock.stripes";
+ public static final int OZONE_CONTAINER_CACHE_LOCK_STRIPES_DEFAULT = 1024;
public static final String OZONE_SCM_BLOCK_SIZE =
"ozone.scm.block.size";
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index b474ac3..a07807b 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -38,6 +38,16 @@
</description>
</property>
<property>
+ <name>ozone.container.cache.lock.stripes</name>
+ <value>1024</value>
+ <tag>PERFORMANCE, CONTAINER, STORAGE</tag>
+ <description>Container DB open is an exclusive operation. We use a stripe
+ lock to guarantee that different threads can open different container DBs
+ concurrently, while for one container DB, only one thread can open it at
+ the same time. This setting controls the lock stripes.
+ </description>
+ </property>
+ <property>
<name>dfs.container.ipc</name>
<value>9859</value>
<tag>OZONE, CONTAINER, MANAGEMENT</tag>
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java
index d2d2901..f4d8f43 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerCache.java
@@ -23,6 +23,7 @@ import java.io.IOException;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
+import com.google.common.util.concurrent.Striped;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.utils.MetadataStore;
import org.apache.hadoop.hdds.utils.MetadataStoreBuilder;
@@ -43,12 +44,14 @@ public final class ContainerCache extends LRUMap {
private final Lock lock = new ReentrantLock();
private static ContainerCache cache;
private static final float LOAD_FACTOR = 0.75f;
+ private final Striped<Lock> rocksDBLock;
/**
* Constructs a cache that holds DBHandle references.
*/
- private ContainerCache(int maxSize, float loadFactor, boolean
+ private ContainerCache(int maxSize, int stripes, float loadFactor, boolean
scanUntilRemovable) {
super(maxSize, loadFactor, scanUntilRemovable);
+ rocksDBLock = Striped.lazyWeakLock(stripes);
}
/**
@@ -63,7 +66,10 @@ public final class ContainerCache extends LRUMap {
if (cache == null) {
int cacheSize = conf.getInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE,
OzoneConfigKeys.OZONE_CONTAINER_CACHE_DEFAULT);
- cache = new ContainerCache(cacheSize, LOAD_FACTOR, true);
+ int stripes = conf.getInt(
+ OzoneConfigKeys.OZONE_CONTAINER_CACHE_LOCK_STRIPES,
+ OzoneConfigKeys.OZONE_CONTAINER_CACHE_LOCK_STRIPES_DEFAULT);
+ cache = new ContainerCache(cacheSize, stripes, LOAD_FACTOR, true);
}
return cache;
}
@@ -117,30 +123,57 @@ public final class ContainerCache extends LRUMap {
throws IOException {
Preconditions.checkState(containerID >= 0,
"Container ID cannot be negative.");
- lock.lock();
+ ReferenceCountedDB db;
+ Lock containerLock = rocksDBLock.get(containerDBPath);
+ containerLock.lock();
try {
- ReferenceCountedDB db = (ReferenceCountedDB) this.get(containerDBPath);
+ lock.lock();
+ try {
+ db = (ReferenceCountedDB) this.get(containerDBPath);
+ if (db != null) {
+ db.incrementReference();
+ return db;
+ }
+ } finally {
+ lock.unlock();
+ }
- if (db == null) {
+ try {
MetadataStore metadataStore =
MetadataStoreBuilder.newBuilder()
- .setDbFile(new File(containerDBPath))
- .setCreateIfMissing(false)
- .setConf(conf)
- .setDBType(containerDBType)
- .build();
+ .setDbFile(new File(containerDBPath))
+ .setCreateIfMissing(false)
+ .setConf(conf)
+ .setDBType(containerDBType)
+ .build();
db = new ReferenceCountedDB(metadataStore, containerDBPath);
- this.put(containerDBPath, db);
+ } catch (Exception e) {
+ LOG.error("Error opening DB. Container:{} ContainerPath:{}",
+ containerID, containerDBPath, e);
+ throw e;
+ }
+
+ lock.lock();
+ try {
+ ReferenceCountedDB currentDB =
+ (ReferenceCountedDB) this.get(containerDBPath);
+ if (currentDB != null) {
+ // increment the reference before returning the object
+ currentDB.incrementReference();
+ // clean the db created in previous step
+ db.cleanup();
+ return currentDB;
+ } else {
+ this.put(containerDBPath, db);
+ // increment the reference before returning the object
+ db.incrementReference();
+ return db;
+ }
+ } finally {
+ lock.unlock();
}
- // increment the reference before returning the object
- db.incrementReference();
- return db;
- } catch (Exception e) {
- LOG.error("Error opening DB. Container:{} ContainerPath:{}",
- containerID, containerDBPath, e);
- throw e;
} finally {
- lock.unlock();
+ containerLock.unlock();
}
}
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java
index 1b9b3d6..fa63cf1 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java
@@ -120,6 +120,7 @@ public class ContainerReader implements Runnable {
return;
}
+ LOG.info("Start to verify containers on volume {}", hddsVolumeRootDir);
for (File scmLoc : scmDir) {
File currentDir = new File(scmLoc, Storage.STORAGE_DIR_CURRENT);
File[] containerTopDirs = currentDir.listFiles();
@@ -144,6 +145,7 @@ public class ContainerReader implements Runnable {
}
}
}
+ LOG.info("Finish verifying containers on volume {}", hddsVolumeRootDir);
}
private void verifyContainerFile(long containerID, File containerFile) {
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
index 62fd5a4..abe0382 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
@@ -163,6 +163,7 @@ public class OzoneContainer {
Iterator<HddsVolume> volumeSetIterator = volumeSet.getVolumesList()
.iterator();
ArrayList<Thread> volumeThreads = new ArrayList<Thread>();
+ long startTime = System.currentTimeMillis();
//TODO: diskchecker should be run before this, to see how disks are.
// And also handle disk failure tolerance need to be added
@@ -183,6 +184,8 @@ public class OzoneContainer {
Thread.currentThread().interrupt();
}
+ LOG.info("Build ContainerSet costs {}s",
+ (System.currentTimeMillis() - startTime) / 1000);
}
/**
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java
index 947a087..2e38990 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java
@@ -31,6 +31,13 @@ import org.junit.Test;
import org.junit.rules.ExpectedException;
import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
/**
@@ -63,6 +70,8 @@ public class TestContainerCache {
conf.setInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE, 2);
ContainerCache cache = ContainerCache.getInstance(conf);
+ cache.clear();
+ Assert.assertEquals(0, cache.size());
File containerDir1 = new File(root, "cont1");
File containerDir2 = new File(root, "cont2");
File containerDir3 = new File(root, "cont3");
@@ -123,4 +132,47 @@ public class TestContainerCache {
thrown.expect(IllegalArgumentException.class);
db5.close();
}
+
+ @Test
+ public void testConcurrentDBGet() throws Exception {
+ File root = new File(testRoot);
+ root.mkdirs();
+ root.deleteOnExit();
+
+ OzoneConfiguration conf = new OzoneConfiguration();
+ conf.setInt(OzoneConfigKeys.OZONE_CONTAINER_CACHE_SIZE, 2);
+ ContainerCache cache = ContainerCache.getInstance(conf);
+ cache.clear();
+ Assert.assertEquals(0, cache.size());
+ File containerDir = new File(root, "cont1");
+ createContainerDB(conf, containerDir);
+ ExecutorService executorService = Executors.newFixedThreadPool(2);
+ Runnable task = () -> {
+ try {
+ ReferenceCountedDB db1 = cache.getDB(1, "RocksDB",
+ containerDir.getPath(), conf);
+ Assert.assertNotNull(db1);
+ } catch (IOException e) {
+ Assert.fail("Should get the DB instance");
+ }
+ };
+ List<Future> futureList = new ArrayList<>();
+ futureList.add(executorService.submit(task));
+ futureList.add(executorService.submit(task));
+ for (Future future: futureList) {
+ try {
+ future.get();
+ } catch (InterruptedException| ExecutionException e) {
+ Assert.fail("Should get the DB instance");
+ }
+ }
+
+ ReferenceCountedDB db = cache.getDB(1, "RocksDB",
+ containerDir.getPath(), conf);
+ db.close();
+ db.close();
+ db.close();
+ Assert.assertEquals(1, cache.size());
+ db.cleanup();
+ }
}
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java
index e1c5f33..15c0236 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java
@@ -22,15 +22,16 @@ import com.google.common.primitives.Longs;
import org.apache.hadoop.conf.StorageUnit;
import org.apache.hadoop.hdds.StringUtils;
import org.apache.hadoop.hdds.client.BlockID;
-import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.ozone.OzoneConsts;
import org.apache.hadoop.ozone.container.common.helpers.BlockData;
import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo;
import org.apache.hadoop.ozone.container.common.impl.ChunkLayOutVersion;
import org.apache.hadoop.ozone.container.common.impl.ContainerSet;
import org.apache.hadoop.ozone.container.common.interfaces.Container;
+import org.apache.hadoop.ozone.container.common.utils.ContainerCache;
import org.apache.hadoop.ozone.container.common.utils.ReferenceCountedDB;
import org.apache.hadoop.ozone.container.common.volume.HddsVolume;
import org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet;
@@ -68,7 +69,7 @@ public class TestContainerReader {
private MutableVolumeSet volumeSet;
private HddsVolume hddsVolume;
private ContainerSet containerSet;
- private ConfigurationSource conf;
+ private OzoneConfiguration conf;
private RoundRobinVolumeChoosingPolicy volumeChoosingPolicy;
@@ -219,4 +220,68 @@ public class TestContainerReader {
keyValueContainerData.getNumPendingDeletionBlocks());
}
}
+
+ @Test
+ public void testMultipleContainerReader() throws Exception {
+ final int volumeNum = 10;
+ StringBuffer datanodeDirs = new StringBuffer();
+ File[] volumeDirs = new File[volumeNum];
+ for (int i = 0; i < volumeNum; i++) {
+ volumeDirs[i] = tempDir.newFolder();
+ datanodeDirs = datanodeDirs.append(volumeDirs[i]).append(",");
+ }
+ conf.set(ScmConfigKeys.HDDS_DATANODE_DIR_KEY,
+ datanodeDirs.toString());
+ MutableVolumeSet volumeSets =
+ new MutableVolumeSet(datanodeId.toString(), conf);
+ ContainerCache cache = ContainerCache.getInstance(conf);
+ cache.clear();
+
+ RoundRobinVolumeChoosingPolicy policy =
+ new RoundRobinVolumeChoosingPolicy();
+
+ final int containerCount = 100;
+ blockCount = containerCount;
+ for (int i = 0; i < containerCount; i++) {
+ KeyValueContainerData keyValueContainerData =
+ new KeyValueContainerData(i, ChunkLayOutVersion.FILE_PER_BLOCK,
+ (long) StorageUnit.GB.toBytes(5), UUID.randomUUID().toString(),
+ datanodeId.toString());
+
+ KeyValueContainer keyValueContainer =
+ new KeyValueContainer(keyValueContainerData,
+ conf);
+ keyValueContainer.create(volumeSets, policy, scmId);
+
+ List<Long> blkNames;
+ if (i % 2 == 0) {
+ blkNames = addBlocks(keyValueContainer, true);
+ markBlocksForDelete(keyValueContainer, true, blkNames, i);
+ } else {
+ blkNames = addBlocks(keyValueContainer, false);
+ markBlocksForDelete(keyValueContainer, false, blkNames, i);
+ }
+ }
+
+ List<HddsVolume> hddsVolumes = volumeSets.getVolumesList();
+ ContainerReader[] containerReaders = new ContainerReader[volumeNum];
+ Thread[] threads = new Thread[volumeNum];
+ for (int i = 0; i < volumeNum; i++) {
+ containerReaders[i] = new ContainerReader(volumeSets,
+ hddsVolumes.get(i), containerSet, conf);
+ threads[i] = new Thread(containerReaders[i]);
+ }
+ long startTime = System.currentTimeMillis();
+ for (int i = 0; i < volumeNum; i++) {
+ threads[i].start();
+ }
+ for (int i = 0; i < volumeNum; i++) {
+ threads[i].join();
+ }
+ System.out.println("Open " + volumeNum + " Volume with " + containerCount +
+ " costs " + (System.currentTimeMillis() - startTime) / 1000 + "s");
+ Assert.assertEquals(containerCount,
+ containerSet.getContainerMap().entrySet().size());
+ Assert.assertEquals(containerCount, cache.size());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 27/39: HDDS-3855. Add upgrade smoketest (#1142)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 6361f4a937d3c4c7b63e94931e940d5af97c43f9
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Fri Jul 17 12:45:03 2020 +0200
HDDS-3855. Add upgrade smoketest (#1142)
(cherry picked from commit 9b13ab67ca433b44a548bd1ee8c9fa4b01250c50)
---
.github/workflows/post-commit.yml | 1 +
.../dist/dev-support/bin/dist-layout-stitching | 1 +
hadoop-ozone/dist/pom.xml | 2 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 39 +++----
hadoop-ozone/dist/src/main/compose/upgrade/.env | 21 ++++
.../dist/src/main/compose/upgrade/README.md | 29 +++++
.../src/main/compose/upgrade/docker-compose.yaml | 127 +++++++++++++++++++++
.../dist/src/main/compose/upgrade/docker-config | 33 ++++++
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 70 ++++++++++++
.../src/main/compose/upgrade/versions/README.md | 15 +++
.../main/compose/upgrade/versions/ozone-0.5.0.sh | 18 +++
.../main/compose/upgrade/versions/ozone-0.6.0.sh | 18 +++
hadoop-ozone/dist/src/shell/upgrade/0.6.0.sh | 23 ++++
.../src/shell/upgrade/0.6.0/01-migrate-scm-db.sh | 24 ++++
14 files changed, 400 insertions(+), 21 deletions(-)
diff --git a/.github/workflows/post-commit.yml b/.github/workflows/post-commit.yml
index e00018a..bac27e4 100644
--- a/.github/workflows/post-commit.yml
+++ b/.github/workflows/post-commit.yml
@@ -150,6 +150,7 @@ jobs:
env:
KEEP_IMAGE: false
OZONE_WITH_COVERAGE: true
+ OZONE_VOLUME_OWNER: 1000
- uses: actions/upload-artifact@master
if: always()
with:
diff --git a/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching b/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
index e1f5c7e..80455a6 100755
--- a/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
+++ b/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
@@ -108,6 +108,7 @@ run cp "${ROOT}/hadoop-ozone/dist/src/shell/hdds/hadoop-config.cmd" "libexec/"
run cp "${ROOT}/hadoop-ozone/dist/src/shell/hdds/hadoop-functions.sh" "libexec/"
run cp "${ROOT}/hadoop-ozone/dist/src/shell/ozone/ozone-config.sh" "libexec/"
run cp -r "${ROOT}/hadoop-ozone/dist/src/shell/shellprofile.d" "libexec/"
+run cp -r "${ROOT}/hadoop-ozone/dist/src/shell/upgrade" "libexec/"
run cp "${ROOT}/hadoop-ozone/dist/src/shell/hdds/hadoop-daemons.sh" "sbin/"
diff --git a/hadoop-ozone/dist/pom.xml b/hadoop-ozone/dist/pom.xml
index 840f628..a766c0a 100644
--- a/hadoop-ozone/dist/pom.xml
+++ b/hadoop-ozone/dist/pom.xml
@@ -28,7 +28,7 @@
<properties>
<file.encoding>UTF-8</file.encoding>
<downloadSources>true</downloadSources>
- <docker.ozone-runner.version>20200420-1</docker.ozone-runner.version>
+ <docker.ozone-runner.version>20200625-1</docker.ozone-runner.version>
</properties>
<build>
diff --git a/hadoop-ozone/dist/src/main/compose/testlib.sh b/hadoop-ozone/dist/src/main/compose/testlib.sh
index 15d1664..56c35c1 100755
--- a/hadoop-ozone/dist/src/main/compose/testlib.sh
+++ b/hadoop-ozone/dist/src/main/compose/testlib.sh
@@ -17,7 +17,6 @@
set -e
COMPOSE_ENV_NAME=$(basename "$COMPOSE_DIR")
-COMPOSE_FILE=$COMPOSE_DIR/docker-compose.yaml
RESULT_DIR=${RESULT_DIR:-"$COMPOSE_DIR/result"}
RESULT_DIR_INSIDE="/tmp/smoketest/$(basename "$COMPOSE_ENV_NAME")/result"
SMOKETEST_DIR_INSIDE="${OZONE_DIR:-/opt/hadoop}/smoketest"
@@ -32,7 +31,7 @@ fi
## @description create results directory, purging any prior data
create_results_dir() {
#delete previous results
- rm -rf "$RESULT_DIR"
+ [[ "${OZONE_KEEP_RESULTS:-}" == "true" ]] || rm -rf "$RESULT_DIR"
mkdir -p "$RESULT_DIR"
#Should be writeable from the docker containers where user is different.
chmod ogu+w "$RESULT_DIR"
@@ -40,9 +39,9 @@ create_results_dir() {
## @description wait until safemode exit (or 180 seconds)
-## @param the docker-compose file
wait_for_safemode_exit(){
- local compose_file=$1
+ # version-dependent
+ : ${OZONE_ADMIN_COMMAND:=admin}
#Reset the timer
SECONDS=0
@@ -51,11 +50,11 @@ wait_for_safemode_exit(){
while [[ $SECONDS -lt 180 ]]; do
#This line checks the safemode status in scm
- local command="ozone admin safemode status"
+ local command="ozone ${OZONE_ADMIN_COMMAND} safemode status"
if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
- status=$(docker-compose -f "${compose_file}" exec -T scm bash -c "kinit -k HTTP/scm@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab && $command" || true)
+ status=$(docker-compose exec -T scm bash -c "kinit -k HTTP/scm@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab && $command" || true)
else
- status=$(docker-compose -f "${compose_file}" exec -T scm bash -c "$command")
+ status=$(docker-compose exec -T scm bash -c "$command")
fi
echo $status
@@ -80,9 +79,9 @@ start_docker_env(){
create_results_dir
export OZONE_SAFEMODE_MIN_DATANODES="${datanode_count}"
- docker-compose -f "$COMPOSE_FILE" --no-ansi down
- if ! { docker-compose -f "$COMPOSE_FILE" --no-ansi up -d --scale datanode="${datanode_count}" \
- && wait_for_safemode_exit "$COMPOSE_FILE"; }; then
+ docker-compose --no-ansi down
+ if ! { docker-compose --no-ansi up -d --scale datanode="${datanode_count}" \
+ && wait_for_safemode_exit ; }; then
OUTPUT_NAME="$COMPOSE_ENV_NAME"
stop_docker_env
return 1
@@ -114,11 +113,11 @@ execute_robot_test(){
OUTPUT_PATH="$RESULT_DIR_INSIDE/${OUTPUT_FILE}"
# shellcheck disable=SC2068
- docker-compose -f "$COMPOSE_FILE" exec -T "$CONTAINER" mkdir -p "$RESULT_DIR_INSIDE" \
- && docker-compose -f "$COMPOSE_FILE" exec -T "$CONTAINER" robot -v OM_SERVICE_ID:"${OM_SERVICE_ID}" -v SECURITY_ENABLED:"${SECURITY_ENABLED}" -v OM_HA_PARAM:"${OM_HA_PARAM}" ${ARGUMENTS[@]} --log NONE -N "$TEST_NAME" --report NONE "${OZONE_ROBOT_OPTS[@]}" --output "$OUTPUT_PATH" "$SMOKETEST_DIR_INSIDE/$TEST"
+ docker-compose exec -T "$CONTAINER" mkdir -p "$RESULT_DIR_INSIDE" \
+ && docker-compose exec -T "$CONTAINER" robot -v OM_SERVICE_ID:"${OM_SERVICE_ID}" -v SECURITY_ENABLED:"${SECURITY_ENABLED}" -v OM_HA_PARAM:"${OM_HA_PARAM}" ${ARGUMENTS[@]} --log NONE -N "$TEST_NAME" --report NONE "${OZONE_ROBOT_OPTS[@]}" --output "$OUTPUT_PATH" "$SMOKETEST_DIR_INSIDE/$TEST"
local -i rc=$?
- FULL_CONTAINER_NAME=$(docker-compose -f "$COMPOSE_FILE" ps | grep "_${CONTAINER}_" | head -n 1 | awk '{print $1}')
+ FULL_CONTAINER_NAME=$(docker-compose ps | grep "_${CONTAINER}_" | head -n 1 | awk '{print $1}')
docker cp "$FULL_CONTAINER_NAME:$OUTPUT_PATH" "$RESULT_DIR/"
copy_daemon_logs
@@ -135,7 +134,7 @@ execute_robot_test(){
## @description Copy any 'out' files for daemon processes to the result dir
copy_daemon_logs() {
local c f
- for c in $(docker-compose -f "$COMPOSE_FILE" ps | grep "^${COMPOSE_ENV_NAME}_" | awk '{print $1}'); do
+ for c in $(docker-compose ps | grep "^${COMPOSE_ENV_NAME}_" | awk '{print $1}'); do
for f in $(docker exec "${c}" ls -1 /var/log/hadoop | grep -F '.out'); do
docker cp "${c}:/var/log/hadoop/${f}" "$RESULT_DIR/"
done
@@ -149,7 +148,7 @@ copy_daemon_logs() {
execute_command_in_container(){
set -e
# shellcheck disable=SC2068
- docker-compose -f "$COMPOSE_FILE" exec -T "$@"
+ docker-compose exec -T "$@"
set +e
}
@@ -157,7 +156,7 @@ execute_command_in_container(){
## @param List of container names, eg datanode_1 datanode_2
stop_containers() {
set -e
- docker-compose -f "$COMPOSE_FILE" --no-ansi stop $@
+ docker-compose --no-ansi stop $@
set +e
}
@@ -166,7 +165,7 @@ stop_containers() {
## @param List of container names, eg datanode_1 datanode_2
start_containers() {
set -e
- docker-compose -f "$COMPOSE_FILE" --no-ansi start $@
+ docker-compose --no-ansi start $@
set +e
}
@@ -185,7 +184,7 @@ wait_for_port(){
while [[ $SECONDS -lt $timeout ]]; do
set +e
- docker-compose -f "${COMPOSE_FILE}" exec -T scm /bin/bash -c "nc -z $host $port"
+ docker-compose exec -T scm /bin/bash -c "nc -z $host $port"
status=$?
set -e
if [ $status -eq 0 ] ; then
@@ -202,9 +201,9 @@ wait_for_port(){
## @description Stops a docker-compose based test environment (with saving the logs)
stop_docker_env(){
- docker-compose -f "$COMPOSE_FILE" --no-ansi logs > "$RESULT_DIR/docker-$OUTPUT_NAME.log"
+ docker-compose --no-ansi logs > "$RESULT_DIR/docker-$OUTPUT_NAME.log"
if [ "${KEEP_RUNNING:-false}" = false ]; then
- docker-compose -f "$COMPOSE_FILE" --no-ansi down
+ docker-compose --no-ansi down
fi
}
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/.env b/hadoop-ozone/dist/src/main/compose/upgrade/.env
new file mode 100644
index 0000000..6f757c5
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/.env
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+HDDS_VERSION=${hdds.version}
+OZONE_RUNNER_VERSION=${docker.ozone-runner.version}
+OZONE_IMAGE=apache/ozone-runner:${docker.ozone-runner.version}
+OZONE_DIR=/opt/hadoop
+OZONE_VOLUME=.
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/README.md b/hadoop-ozone/dist/src/main/compose/upgrade/README.md
new file mode 100644
index 0000000..2a832f4
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/README.md
@@ -0,0 +1,29 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Compose file for upgrade
+
+This directory contains a sample cluster definition and script for
+testing upgrade from previous version to the current one.
+
+Data for each container is persisted in mounted volume (by default it's
+`data` under the `compose/upgrade` directory, but can be overridden via
+`OZONE_VOLUME` environment variable).
+
+Prior version is run using an official `apache/ozone` image, while the
+current version is run with the `ozone-runner` image using locally built
+source code.
+
+Currently the test script only supports a single version upgrade (eg.
+from 0.5.0 to 0.6.0).
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/upgrade/docker-compose.yaml
new file mode 100644
index 0000000..ae52759
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/docker-compose.yaml
@@ -0,0 +1,127 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.4"
+
+# reusable fragments (see https://docs.docker.com/compose/compose-file/#extension-fields)
+x-common-config:
+ &common-config
+ env_file:
+ - docker-config
+ image: ${OZONE_IMAGE}
+
+x-replication:
+ &replication
+ OZONE-SITE.XML_ozone.replication: ${OZONE_REPLICATION_FACTOR:-1}
+
+x-datanode:
+ &datanode
+ command: ["ozone","datanode"]
+ <<: *common-config
+ environment:
+ <<: *replication
+ ports:
+ - 9864
+ - 9882
+
+services:
+ dn1:
+ <<: *datanode
+ networks:
+ net:
+ ipv4_address: 10.9.0.11
+ volumes:
+ - ${OZONE_VOLUME}/dn1:/data
+ - ../..:${OZONE_DIR}
+ dn2:
+ <<: *datanode
+ networks:
+ net:
+ ipv4_address: 10.9.0.12
+ volumes:
+ - ${OZONE_VOLUME}/dn2:/data
+ - ../..:${OZONE_DIR}
+ dn3:
+ <<: *datanode
+ networks:
+ net:
+ ipv4_address: 10.9.0.13
+ volumes:
+ - ${OZONE_VOLUME}/dn3:/data
+ - ../..:${OZONE_DIR}
+ om:
+ command: ["ozone","om"]
+ <<: *common-config
+ environment:
+ ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
+ <<: *replication
+ networks:
+ net:
+ ipv4_address: 10.9.0.14
+ ports:
+ - 9874:9874
+ - 9862:9862
+ volumes:
+ - ${OZONE_VOLUME}/om:/data
+ - ../..:${OZONE_DIR}
+ recon:
+ command: ["ozone","recon"]
+ <<: *common-config
+ environment:
+ <<: *replication
+ networks:
+ net:
+ ipv4_address: 10.9.0.15
+ ports:
+ - 9888:9888
+ volumes:
+ - ${OZONE_VOLUME}/recon:/data
+ - ../..:${OZONE_DIR}
+ s3g:
+ command: ["ozone","s3g"]
+ <<: *common-config
+ environment:
+ <<: *replication
+ networks:
+ net:
+ ipv4_address: 10.9.0.16
+ ports:
+ - 9878:9878
+ volumes:
+ - ${OZONE_VOLUME}/s3g:/data
+ - ../..:${OZONE_DIR}
+ scm:
+ command: ["ozone","scm"]
+ <<: *common-config
+ environment:
+ ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
+ OZONE-SITE.XML_hdds.scm.safemode.min.datanode: ${OZONE_SAFEMODE_MIN_DATANODES:-1}
+ <<: *replication
+ networks:
+ net:
+ ipv4_address: 10.9.0.17
+ ports:
+ - 9876:9876
+ volumes:
+ - ${OZONE_VOLUME}/scm:/data
+ - ../..:${OZONE_DIR}
+
+networks:
+ net:
+ driver: bridge
+ ipam:
+ config:
+ - subnet: 10.9.0.0/16
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/docker-config b/hadoop-ozone/dist/src/main/compose/upgrade/docker-config
new file mode 100644
index 0000000..ecc994b
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/docker-config
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
+CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
+OZONE-SITE.XML_ozone.om.address=om
+OZONE-SITE.XML_ozone.om.http-address=om:9874
+OZONE-SITE.XML_ozone.scm.container.size=1GB
+OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1
+OZONE-SITE.XML_ozone.scm.names=scm
+OZONE-SITE.XML_ozone.scm.datanode.id.dir=/data
+OZONE-SITE.XML_ozone.scm.block.client.address=scm
+OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
+OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
+OZONE-SITE.XML_ozone.scm.client.address=scm
+OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
+OZONE-SITE.XML_ozone.recon.address=recon:9891
+OZONE-SITE.XML_recon.om.snapshot.task.interval.delay=1m
+
+no_proxy=om,scm,s3g,kdc,localhost,127.0.0.1
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/test.sh b/hadoop-ozone/dist/src/main/compose/upgrade/test.sh
new file mode 100644
index 0000000..0babd17
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/test.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+
+: "${OZONE_REPLICATION_FACTOR:=3}"
+: "${OZONE_UPGRADE_FROM:="0.5.0"}"
+: "${OZONE_UPGRADE_TO:="0.6.0"}"
+: "${OZONE_VOLUME:="${COMPOSE_DIR}/data"}"
+
+export OZONE_VOLUME
+
+mkdir -p "${OZONE_VOLUME}"/{dn1,dn2,dn3,om,recon,s3g,scm}
+mkdir -p "${OZONE_VOLUME}/debug"
+
+if [[ -n "${OZONE_VOLUME_OWNER}" ]]; then
+ current_user=$(whoami)
+ if [[ "${OZONE_VOLUME_OWNER}" != "${current_user}" ]]; then
+ chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}" \
+ || sudo chown -R "${OZONE_VOLUME_OWNER}" "${OZONE_VOLUME}"
+ fi
+fi
+
+# define version-specifics
+export OZONE_DIR=/opt/ozone
+export OZONE_IMAGE="apache/ozone:${OZONE_UPGRADE_FROM}"
+# shellcheck source=/dev/null
+source "${COMPOSE_DIR}/versions/ozone-${OZONE_UPGRADE_FROM}.sh"
+# shellcheck source=/dev/null
+source "${COMPOSE_DIR}/../testlib.sh"
+
+# prepare pre-upgrade cluster
+start_docker_env
+execute_robot_test scm topology/loaddata.robot
+stop_docker_env
+
+# run upgrade scripts
+SCRIPT_DIR=../../libexec/upgrade
+[[ -f "${SCRIPT_DIR}/${OZONE_UPGRADE_TO}.sh" ]] && "${SCRIPT_DIR}/${OZONE_UPGRADE_TO}.sh"
+
+# update version-specifics
+export OZONE_DIR=/opt/hadoop
+unset OZONE_IMAGE # use apache/ozone-runner defined in docker-compose.yaml
+# shellcheck source=/dev/null
+source "${COMPOSE_DIR}/versions/ozone-${OZONE_UPGRADE_TO}.sh"
+# shellcheck source=/dev/null
+source "${COMPOSE_DIR}/../testlib.sh"
+
+# re-start cluster with new version and check after upgrade
+export OZONE_KEEP_RESULTS=true
+start_docker_env
+execute_robot_test scm topology/readdata.robot
+stop_docker_env
+
+generate_report
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/versions/README.md b/hadoop-ozone/dist/src/main/compose/upgrade/versions/README.md
new file mode 100644
index 0000000..c662c2f
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/versions/README.md
@@ -0,0 +1,15 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+The scripts in this directory define version-specific behavior required for [`testlib.sh`](../../testlib.sh). For example the `ozone admin` command was renamed from `ozone scmcli` in 0.6.0.
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.5.0.sh b/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.5.0.sh
new file mode 100644
index 0000000..68f6c75
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.5.0.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export OZONE_ADMIN_COMMAND=scmcli
diff --git a/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.6.0.sh b/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.6.0.sh
new file mode 100644
index 0000000..bab1dba
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/compose/upgrade/versions/ozone-0.6.0.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export OZONE_ADMIN_COMMAND=admin
diff --git a/hadoop-ozone/dist/src/shell/upgrade/0.6.0.sh b/hadoop-ozone/dist/src/shell/upgrade/0.6.0.sh
new file mode 100755
index 0000000..58c78db
--- /dev/null
+++ b/hadoop-ozone/dist/src/shell/upgrade/0.6.0.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+: "${SCM_DIR:="${OZONE_VOLUME}/scm"}"
+: "${OZONE_RUNNER_VERSION:="20200625-1"}"
+
+docker run --rm -v "${SCM_DIR}":/scm -v "${SCRIPT_DIR}/0.6.0":/upgrade -w /scm/metadata apache/ozone-runner:"${OZONE_RUNNER_VERSION}" /upgrade/01-migrate-scm-db.sh
diff --git a/hadoop-ozone/dist/src/shell/upgrade/0.6.0/01-migrate-scm-db.sh b/hadoop-ozone/dist/src/shell/upgrade/0.6.0/01-migrate-scm-db.sh
new file mode 100755
index 0000000..dee6768
--- /dev/null
+++ b/hadoop-ozone/dist/src/shell/upgrade/0.6.0/01-migrate-scm-db.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+echo Running upgrade script for HDDS-3499
+
+ldb --db=scm.db create_column_family containers
+ldb --db=scm.db create_column_family pipelines
+
+ldb --db=scm-container.db --key_hex --value_hex dump | ldb --db=scm.db --key_hex --value_hex --column_family=containers load
+ldb --db=scm-pipeline.db --key_hex --value_hex dump | ldb --db=scm.db --key_hex --value_hex --column_family=pipelines load
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 08/39: HDDS-3062. Fix
TestOzoneRpcClientAbstract.testListVolume. (#1194)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 2e529a9490fd02fbc6555b93b827baaaf3ae5a4b
Author: Lokesh Jain <lj...@apache.org>
AuthorDate: Tue Jul 14 18:52:24 2020 +0530
HDDS-3062. Fix TestOzoneRpcClientAbstract.testListVolume. (#1194)
(cherry picked from commit 8bb452f37a8ae977131a94059b246201cc5d51de)
---
.../org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java | 3 ---
1 file changed, 3 deletions(-)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
index fb17882..f0ffee3 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
@@ -1234,9 +1234,6 @@ public abstract class TestOzoneRpcClientAbstract {
Assert.assertEquals(toKeyName, key.getName());
}
- // Listing all volumes in the cluster feature has to be fixed after HDDS-357.
- // TODO: fix this
- @Ignore
@Test
public void testListVolume() throws IOException {
String volBase = "vol-" + RandomStringUtils.randomNumeric(3);
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 06/39: HDDS-3861. Fix handlePipelineFailure throw
exception if role is follower (#1122)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 29e26b0f14f7f2105799938f5cdee340c6e5b1ac
Author: runzhiwang <51...@users.noreply.github.com>
AuthorDate: Tue Jul 14 11:59:03 2020 +0800
HDDS-3861. Fix handlePipelineFailure throw exception if role is follower (#1122)
(cherry picked from commit 537172ee2cdb343359082dfc0c23c01068d4f24e)
---
.../common/transport/server/ratis/XceiverServerRatis.java | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
index 6c98e81..3e6ac10 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
@@ -552,6 +552,14 @@ public final class XceiverServerRatis implements XceiverServerSpi {
msg = datanode + " is in candidate state for " +
roleInfoProto.getCandidateInfo().getLastLeaderElapsedTimeMs() + "ms";
break;
+ case FOLLOWER:
+ msg = datanode + " closes pipeline when installSnapshot from leader " +
+ "because leader snapshot doesn't contain any data to replay, " +
+ "all the log entries prior to the snapshot might have been purged." +
+ "So follower should not try to install snapshot from leader but" +
+ "can close the pipeline here. It's in follower state for " +
+ roleInfoProto.getRoleElapsedTimeMs() + "ms";
+ break;
case LEADER:
StringBuilder sb = new StringBuilder();
sb.append(datanode).append(" has not seen follower/s");
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 03/39: HDDS-3937. Update jquery to v3.5.1 (#1177)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit f58edcf6b70cd243aeffa21da9cd99fa4852dd5f
Author: Vivek Ratnavel Subramanian <vi...@gmail.com>
AuthorDate: Sun Jul 12 23:39:24 2020 -0700
HDDS-3937. Update jquery to v3.5.1 (#1177)
(cherry picked from commit 7054b4ba8ed886269e4a822133917564181ae8d4)
---
LICENSE.txt | 4 ++--
hadoop-hdds/docs/pom.xml | 2 +-
hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html | 2 +-
hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js | 2 --
hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js | 2 ++
.../framework/src/main/resources/webapps/static/jquery-3.4.1.min.js | 2 --
.../framework/src/main/resources/webapps/static/jquery-3.5.1.min.js | 2 ++
hadoop-hdds/pom.xml | 2 +-
hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html | 2 +-
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 2 +-
.../ozone-manager/src/main/resources/webapps/ozoneManager/index.html | 2 +-
hadoop-ozone/pom.xml | 2 +-
hadoop-ozone/s3gateway/src/main/resources/browser.html | 2 +-
hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html | 2 +-
14 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/LICENSE.txt b/LICENSE.txt
index 98fd68d..e7a3522 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -235,5 +235,5 @@ hadoop-hdds/framework/src/main/resources/webapps/static/angular-route-1.7.9.min.
hadoop-hdds/framework/src/main/resources/webapps/static/angular-nvd3-1.0.9.min.js
hadoop-hdds/framework/src/main/resources/webapps/static/angular-1.7.9.min.js
-hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.4.1.min.js
-hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js
+hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.5.1.min.js
+hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js
diff --git a/hadoop-hdds/docs/pom.xml b/hadoop-hdds/docs/pom.xml
index 8c1b622..dd4e5b7 100644
--- a/hadoop-hdds/docs/pom.xml
+++ b/hadoop-hdds/docs/pom.xml
@@ -55,7 +55,7 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<excludes>
<exclude>static/slides/*</exclude>
<exclude>themes/ozonedoc/static/js/bootstrap.min.js</exclude>
- <exclude>themes/ozonedoc/static/js/jquery-3.4.1.min.js</exclude>
+ <exclude>themes/ozonedoc/static/js/jquery-3.5.1.min.js</exclude>
<exclude>themes/ozonedoc/static/css/bootstrap-theme.min.css
</exclude>
<exclude>themes/ozonedoc/static/css/bootstrap.min.css.map</exclude>
diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html
index 0e5ca0f..20bf76e 100644
--- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html
+++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/footer.html
@@ -17,6 +17,6 @@
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
-<script src="{{ "js/jquery-3.4.1.min.js" | relURL}}"></script>
+<script src="{{ "js/jquery-3.5.1.min.js" | relURL}}"></script>
<script src="{{ "js/ozonedoc.js" | relURL}}"></script>
<script src="{{ "js/bootstrap.min.js" | relURL}}"></script>
diff --git a/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js b/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js
deleted file mode 100644
index a1c07fd..0000000
--- a/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.4.1.min.js
+++ /dev/null
@@ -1,2 +0,0 @@
-/*! jQuery v3.4.1 | (c) JS Foundation and other contributors | jquery.org/license */
-!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(C,e){"use strict";var t=[],E=C.document,r=Object.getPrototypeOf,s=t.slice,g=t.concat,u=t.push,i=t.indexOf,n={},o=n.toString,v=n.hasOwnProperty,a=v.toString,l=a.call(Object),y={},m=function(e){return"function"==typeof e& [...]
diff --git a/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js b/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js
new file mode 100644
index 0000000..b061403
--- /dev/null
+++ b/hadoop-hdds/docs/themes/ozonedoc/static/js/jquery-3.5.1.min.js
@@ -0,0 +1,2 @@
+/*! jQuery v3.5.1 | (c) JS Foundation and other contributors | jquery.org/license */
+!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(C,e){"use strict";var t=[],r=Object.getPrototypeOf,s=t.slice,g=t.flat?function(e){return t.flat.call(e)}:function(e){return t.concat.apply([],e)},u=t.push,i=t.indexOf,n={},o=n.toString,v=n.hasOwnProperty,a=v.toString,l= [...]
diff --git a/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.4.1.min.js b/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.4.1.min.js
deleted file mode 100644
index a1c07fd..0000000
--- a/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.4.1.min.js
+++ /dev/null
@@ -1,2 +0,0 @@
-/*! jQuery v3.4.1 | (c) JS Foundation and other contributors | jquery.org/license */
-!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(C,e){"use strict";var t=[],E=C.document,r=Object.getPrototypeOf,s=t.slice,g=t.concat,u=t.push,i=t.indexOf,n={},o=n.toString,v=n.hasOwnProperty,a=v.toString,l=a.call(Object),y={},m=function(e){return"function"==typeof e& [...]
diff --git a/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.5.1.min.js b/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.5.1.min.js
new file mode 100644
index 0000000..b061403
--- /dev/null
+++ b/hadoop-hdds/framework/src/main/resources/webapps/static/jquery-3.5.1.min.js
@@ -0,0 +1,2 @@
+/*! jQuery v3.5.1 | (c) JS Foundation and other contributors | jquery.org/license */
+!function(e,t){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error("jQuery requires a window with a document");return t(e)}:t(e)}("undefined"!=typeof window?window:this,function(C,e){"use strict";var t=[],r=Object.getPrototypeOf,s=t.slice,g=t.flat?function(e){return t.flat.call(e)}:function(e){return t.concat.apply([],e)},u=t.push,i=t.indexOf,n={},o=n.toString,v=n.hasOwnProperty,a=v.toString,l= [...]
diff --git a/hadoop-hdds/pom.xml b/hadoop-hdds/pom.xml
index 8dc3833..4351672 100644
--- a/hadoop-hdds/pom.xml
+++ b/hadoop-hdds/pom.xml
@@ -263,7 +263,7 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<exclude>src/main/resources/webapps/static/nvd3-1.8.5.min.css</exclude>
<exclude>src/main/resources/webapps/static/nvd3-1.8.5.min.js.map</exclude>
<exclude>src/main/resources/webapps/static/nvd3-1.8.5.min.js</exclude>
- <exclude>src/main/resources/webapps/static/jquery-3.4.1.min.js</exclude>
+ <exclude>src/main/resources/webapps/static/jquery-3.5.1.min.js</exclude>
<exclude>src/main/resources/webapps/static/bootstrap-3.4.1/**</exclude>
<exclude>src/test/resources/additionalfields.container</exclude>
<exclude>src/test/resources/incorrect.checksum.container</exclude>
diff --git a/hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html b/hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html
index d586856..5122cae 100644
--- a/hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html
+++ b/hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html
@@ -63,7 +63,7 @@
</div><!-- /.container -->
-<script src="static/jquery-3.4.1.min.js"></script>
+<script src="static/jquery-3.5.1.min.js"></script>
<script src="static/angular-1.7.9.min.js"></script>
<script src="static/angular-route-1.7.9.min.js"></script>
<script src="static/d3-3.5.17.min.js"></script>
diff --git a/hadoop-ozone/dist/src/main/license/bin/LICENSE.txt b/hadoop-ozone/dist/src/main/license/bin/LICENSE.txt
index 54c8fe2..df63c6e 100644
--- a/hadoop-ozone/dist/src/main/license/bin/LICENSE.txt
+++ b/hadoop-ozone/dist/src/main/license/bin/LICENSE.txt
@@ -424,7 +424,7 @@ bootstrap.min.js
angular-route-1.7.9.min.js
angular-nvd3-1.0.9.min.js
angular-1.7.9.min.js
-jquery-3.4.1.min.js
+jquery-3.5.1.min.js
--------------------------------------------------------------------------------
recon server uses a huge number of javascript and css dependencies. See the
diff --git a/hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/index.html b/hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/index.html
index 5fb6e44..8a5297b 100644
--- a/hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/index.html
+++ b/hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/index.html
@@ -57,7 +57,7 @@
<ng-view></ng-view>
</div><!-- /.container -->
-<script src="static/jquery-3.4.1.min.js"></script>
+<script src="static/jquery-3.5.1.min.js"></script>
<script src="static/angular-1.7.9.min.js"></script>
<script src="static/angular-route-1.7.9.min.js"></script>
<script src="static/d3-3.5.17.min.js"></script>
diff --git a/hadoop-ozone/pom.xml b/hadoop-ozone/pom.xml
index 9455fd2..1b28ea4 100644
--- a/hadoop-ozone/pom.xml
+++ b/hadoop-ozone/pom.xml
@@ -291,7 +291,7 @@
<exclude>webapps/static/angular-route-1.7.9.min.js</exclude>
<exclude>webapps/static/bootstrap-3.4.1/**</exclude>
<exclude>webapps/static/d3-3.5.17.min.js</exclude>
- <exclude>webapps/static/jquery-3.4.1.min.js</exclude>
+ <exclude>webapps/static/jquery-3.5.1.min.js</exclude>
<exclude>webapps/static/jquery.dataTables.min.js</exclude>
<exclude>webapps/static/nvd3-1.8.5.min.css.map</exclude>
<exclude>webapps/static/nvd3-1.8.5.min.css</exclude>
diff --git a/hadoop-ozone/s3gateway/src/main/resources/browser.html b/hadoop-ozone/s3gateway/src/main/resources/browser.html
index 0405b17..4e6a00b 100644
--- a/hadoop-ozone/s3gateway/src/main/resources/browser.html
+++ b/hadoop-ozone/s3gateway/src/main/resources/browser.html
@@ -138,7 +138,7 @@ permissions and limitations under the License.
</html>
-<script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
+<script src="https://code.jquery.com/jquery-3.5.1.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/bootbox.js/4.4.0/bootbox.min.js"></script>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.207.0.min.js"></script>
diff --git a/hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html b/hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html
index b20bf35..81158f1 100644
--- a/hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html
+++ b/hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html
@@ -75,7 +75,7 @@
</p>
</div><!-- /.container -->
-<script src="jquery-3.4.1.min.js"></script>
+<script src="jquery-3.5.1.min.js"></script>
<script src="bootstrap-3.4.1/js/bootstrap.min.js"></script>
<script src="s3g.js"></script>
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 26/39: HDDS-3964. Ratis config key mismatch (#1204)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 075423bc5f4fca68ab92acaeedc68481bf185097
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Fri Jul 17 08:20:19 2020 +0200
HDDS-3964. Ratis config key mismatch (#1204)
(cherry picked from commit f15b011ec6ae80fb9eb03dd0e1bdaf6b730fd033)
---
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 24 +++++++---------
.../hadoop/hdds/ratis/conf/RatisClientConfig.java | 6 ++--
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 7 +++--
.../hdds/conf/DatanodeRatisServerConfig.java | 32 +++++++---------------
.../hadoop/fs/ozone/contract/OzoneContract.java | 17 ++++++------
.../ozone/contract/rooted/RootedOzoneContract.java | 17 ++++++------
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 2 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 16 +++++------
.../rpc/TestBlockOutputStreamWithFailures.java | 28 ++++++++++---------
...estBlockOutputStreamWithFailuresFlushDelay.java | 28 ++++++++++---------
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 27 +++++++++---------
.../TestContainerStateMachineFailureOnRead.java | 27 +++++++++---------
.../rpc/TestContainerStateMachineFailures.java | 27 +++++++++---------
.../client/rpc/TestDeleteWithSlowFollower.java | 28 ++++++++++---------
.../client/rpc/TestFailureHandlingByClient.java | 28 ++++++++++---------
.../rpc/TestFailureHandlingByClientFlushDelay.java | 28 ++++++++++---------
.../rpc/TestMultiBlockWritesWithDnFailures.java | 28 ++++++++++---------
.../client/rpc/TestValidateBCSIDOnRestart.java | 28 +++++++++----------
.../ozone/client/rpc/TestWatchForCommit.java | 28 ++++++++++---------
.../hadoop/ozone/freon/TestDataValidate.java | 17 ++++++------
.../ozone/freon/TestFreonWithDatanodeRestart.java | 17 ++++++------
.../ozone/freon/TestFreonWithPipelineDestroy.java | 17 ++++++------
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 17 ++++++------
23 files changed, 246 insertions(+), 248 deletions(-)
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java
index 47523bc..505b6c9 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java
@@ -75,13 +75,6 @@ public final class RatisHelper {
// Prefix for Ratis Server GRPC and Ratis client conf.
public static final String HDDS_DATANODE_RATIS_PREFIX_KEY = "hdds.ratis";
- private static final String RAFT_SERVER_PREFIX_KEY = "raft.server";
- public static final String HDDS_DATANODE_RATIS_SERVER_PREFIX_KEY =
- HDDS_DATANODE_RATIS_PREFIX_KEY + "." + RAFT_SERVER_PREFIX_KEY;
- public static final String HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY =
- HDDS_DATANODE_RATIS_PREFIX_KEY + "." + RaftClientConfigKeys.PREFIX;
- public static final String HDDS_DATANODE_RATIS_GRPC_PREFIX_KEY =
- HDDS_DATANODE_RATIS_PREFIX_KEY + "." + GrpcConfigKeys.PREFIX;
private static final Class[] NO_RETRY_EXCEPTIONS =
new Class[] {NotReplicatedException.class, GroupMismatchException.class,
@@ -227,7 +220,7 @@ public final class RatisHelper {
}
/**
- * Set all the properties matching with regex
+ * Set all client properties matching with regex
* {@link RatisHelper#HDDS_DATANODE_RATIS_PREFIX_KEY} in
* ozone configuration object and configure it to RaftProperties.
* @param ozoneConf
@@ -237,23 +230,26 @@ public final class RatisHelper {
RaftProperties raftProperties) {
// As for client we do not require server and grpc server/tls. exclude them.
- Map<String, String> ratisClientConf = ozoneConf.getPropsWithPrefix(
- StringUtils.appendIfNotPresent(HDDS_DATANODE_RATIS_PREFIX_KEY, '.'));
+ Map<String, String> ratisClientConf =
+ getDatanodeRatisPrefixProps(ozoneConf);
ratisClientConf.forEach((key, val) -> {
- if (key.startsWith(RaftClientConfigKeys.PREFIX) || isGrpcClientConfig(
- key)) {
+ if (isClientConfig(key) || isGrpcClientConfig(key)) {
raftProperties.set(key, val);
}
});
}
+ private static boolean isClientConfig(String key) {
+ return key.startsWith(RaftClientConfigKeys.PREFIX);
+ }
+
private static boolean isGrpcClientConfig(String key) {
return key.startsWith(GrpcConfigKeys.PREFIX) && !key
.startsWith(GrpcConfigKeys.TLS.PREFIX) && !key
.startsWith(GrpcConfigKeys.Server.PREFIX);
}
/**
- * Set all the properties matching with prefix
+ * Set all server properties matching with prefix
* {@link RatisHelper#HDDS_DATANODE_RATIS_PREFIX_KEY} in
* ozone configuration object and configure it to RaftProperties.
* @param ozoneConf
@@ -266,7 +262,7 @@ public final class RatisHelper {
getDatanodeRatisPrefixProps(ozoneConf);
ratisServerConf.forEach((key, val) -> {
// Exclude ratis client configuration.
- if (!key.startsWith(RaftClientConfigKeys.PREFIX)) {
+ if (!isClientConfig(key)) {
raftProperties.set(key, val);
}
});
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java
index 18d174d..b88583d 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/conf/RatisClientConfig.java
@@ -29,7 +29,7 @@ import static org.apache.hadoop.hdds.conf.ConfigTag.PERFORMANCE;
/**
* Configuration related to Ratis Client. This is the config used in creating
- * RaftClient creation.
+ * RaftClient.
*/
@ConfigGroup(prefix = RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY)
public class RatisClientConfig {
@@ -38,7 +38,9 @@ public class RatisClientConfig {
* Configurations which will be set in RaftProperties. RaftProperties is a
* configuration object for Ratis client.
*/
- @ConfigGroup(prefix = RaftClientConfigKeys.PREFIX)
+ @ConfigGroup(prefix =
+ RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY + "." +
+ RaftClientConfigKeys.PREFIX)
public static class RaftConfig {
@Config(key = "async.outstanding-requests.max",
defaultValue = "32",
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisGrpcConfig.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisGrpcConfig.java
index ed036fd..443b879 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisGrpcConfig.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisGrpcConfig.java
@@ -18,15 +18,18 @@
package org.apache.hadoop.hdds.conf;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+
import static org.apache.hadoop.hdds.conf.ConfigTag.CLIENT;
import static org.apache.hadoop.hdds.conf.ConfigTag.OZONE;
import static org.apache.hadoop.hdds.conf.ConfigTag.PERFORMANCE;
-import static org.apache.hadoop.hdds.ratis.RatisHelper.HDDS_DATANODE_RATIS_GRPC_PREFIX_KEY;
+import static org.apache.hadoop.hdds.ratis.RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY;
/**
* Ratis Grpc Config Keys.
*/
-@ConfigGroup(prefix = HDDS_DATANODE_RATIS_GRPC_PREFIX_KEY)
+@ConfigGroup(prefix = HDDS_DATANODE_RATIS_PREFIX_KEY + "."
+ + GrpcConfigKeys.PREFIX)
public class DatanodeRatisGrpcConfig {
@Config(key = "message.size.max",
defaultValue = "32MB",
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java
index 39f58bb..b0034ee 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java
@@ -18,36 +18,24 @@
package org.apache.hadoop.hdds.conf;
+import org.apache.ratis.server.RaftServerConfigKeys;
+
import java.time.Duration;
import static org.apache.hadoop.hdds.conf.ConfigTag.DATANODE;
import static org.apache.hadoop.hdds.conf.ConfigTag.OZONE;
import static org.apache.hadoop.hdds.conf.ConfigTag.PERFORMANCE;
import static org.apache.hadoop.hdds.conf.ConfigTag.RATIS;
-import static org.apache.hadoop.hdds.ratis.RatisHelper.HDDS_DATANODE_RATIS_SERVER_PREFIX_KEY;
+import static org.apache.hadoop.hdds.ratis.RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY;
/**
* Datanode Ratis server Configuration.
*/
-@ConfigGroup(prefix = HDDS_DATANODE_RATIS_SERVER_PREFIX_KEY)
+@ConfigGroup(prefix = HDDS_DATANODE_RATIS_PREFIX_KEY + "."
+ + RaftServerConfigKeys.PREFIX)
public class DatanodeRatisServerConfig {
- private static final String RATIS_SERVER_REQUEST_TIMEOUT_KEY =
- "rpc.request.timeout";
-
- private static final String RATIS_SERVER_WATCH_REQUEST_TIMEOUT_KEY =
- "watch.timeout";
-
- private static final String RATIS_SERVER_NO_LEADER_TIMEOUT_KEY =
- "Notification.no-leader.timeout";
-
- private static final String RATIS_FOLLOWER_SLOWNESS_TIMEOUT_KEY =
- "rpcslowness.timeout";
-
- private static final String RATIS_LEADER_NUM_PENDING_REQUESTS_KEY =
- "write.element-limit";
-
- @Config(key = RATIS_SERVER_REQUEST_TIMEOUT_KEY,
+ @Config(key = "rpc.request.timeout",
defaultValue = "60s",
type = ConfigType.TIME,
tags = {OZONE, DATANODE, RATIS},
@@ -64,7 +52,7 @@ public class DatanodeRatisServerConfig {
this.requestTimeOut = duration.toMillis();
}
- @Config(key = RATIS_SERVER_WATCH_REQUEST_TIMEOUT_KEY,
+ @Config(key = "watch.timeout",
defaultValue = "180s",
type = ConfigType.TIME,
tags = {OZONE, DATANODE, RATIS},
@@ -82,7 +70,7 @@ public class DatanodeRatisServerConfig {
this.watchTimeOut = duration.toMillis();
}
- @Config(key = RATIS_SERVER_NO_LEADER_TIMEOUT_KEY,
+ @Config(key = "notification.no-leader.timeout",
defaultValue = "300s",
type = ConfigType.TIME,
tags = {OZONE, DATANODE, RATIS},
@@ -100,7 +88,7 @@ public class DatanodeRatisServerConfig {
this.noLeaderTimeout = duration.toMillis();
}
- @Config(key = RATIS_FOLLOWER_SLOWNESS_TIMEOUT_KEY,
+ @Config(key = "rpcslowness.timeout",
defaultValue = "300s",
type = ConfigType.TIME,
tags = {OZONE, DATANODE, RATIS},
@@ -118,7 +106,7 @@ public class DatanodeRatisServerConfig {
this.followerSlownessTimeout = duration.toMillis();
}
- @Config(key = RATIS_LEADER_NUM_PENDING_REQUESTS_KEY,
+ @Config(key = "write.element-limit",
defaultValue = "1024",
type = ConfigType.INT,
tags = {OZONE, DATANODE, RATIS, PERFORMANCE},
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/OzoneContract.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/OzoneContract.java
index 7335a93..e832dd7 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/OzoneContract.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/OzoneContract.java
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.contract.AbstractFSContract;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.apache.hadoop.ozone.OzoneConsts;
@@ -71,14 +71,13 @@ class OzoneContract extends AbstractFSContract {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
conf.addResource(CONTRACT_XML);
cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(5).build();
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java
index c90a7ba..cd68929 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.contract.AbstractFSContract;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.apache.hadoop.ozone.OzoneConsts;
@@ -68,14 +68,13 @@ class RootedOzoneContract extends AbstractFSContract {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
conf.addResource(CONTRACT_XML);
cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(5).build();
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java
index 2bb1fb1..6f58eae 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java
@@ -63,7 +63,7 @@ public class TestNodeFailure {
DatanodeRatisServerConfig ratisServerConfig =
conf.getObject(DatanodeRatisServerConfig.class);
ratisServerConfig.setFollowerSlownessTimeout(Duration.ofSeconds(10));
- ratisServerConfig.setNoLeaderTimeout(Duration.ofSeconds(10));
+ ratisServerConfig.setNoLeaderTimeout(Duration.ofMinutes(5));
conf.setFromObject(ratisServerConfig);
conf.set(HddsConfigKeys.HDDS_PIPELINE_REPORT_INTERVAL, "2s");
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
index 087376b..7da9d11 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
@@ -22,7 +22,7 @@ import org.apache.hadoop.hdds.HddsConfigKeys;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
import org.apache.hadoop.hdds.scm.XceiverClientRatis;
import org.apache.hadoop.hdds.scm.XceiverClientSpi;
@@ -97,14 +97,12 @@ public class Test2WayCommitInRatis {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
conf.setQuietMode(false);
cluster = MiniOzoneCluster.newBuilder(conf)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java
index 40756b0..37094d5 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java
@@ -21,7 +21,7 @@ import org.apache.hadoop.hdds.client.ReplicationFactor;
import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.XceiverClientRatis;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
@@ -108,19 +108,21 @@ public class TestBlockOutputStreamWithFailures {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
conf.setBoolean(OZONE_CLIENT_STREAM_BUFFER_FLUSH_DELAY, false);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(7)
.setTotalPipelineNumLimit(10).setBlockSize(blockSize)
.setChunkSize(chunkSize).setStreamBufferFlushSize(flushSize)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailuresFlushDelay.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailuresFlushDelay.java
index 6fa54ed..3462e50 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailuresFlushDelay.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailuresFlushDelay.java
@@ -22,7 +22,7 @@ import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.XceiverClientRatis;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
@@ -102,18 +102,20 @@ public class TestBlockOutputStreamWithFailuresFlushDelay {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(7)
.setTotalPipelineNumLimit(10).setBlockSize(blockSize)
.setChunkSize(chunkSize).setStreamBufferFlushSize(flushSize)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
index b9b7e5e..c3eca6a 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
@@ -23,7 +23,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
import org.apache.hadoop.hdds.scm.XceiverClientRatis;
import org.apache.hadoop.hdds.scm.XceiverClientReply;
@@ -112,18 +112,19 @@ public class TestCommitWatcher {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
conf.set(OzoneConfigKeys.OZONE_CLIENT_CHECKSUM_TYPE, "NONE");
conf.setQuietMode(false);
conf.setStorageSize(OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE, 4,
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailureOnRead.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailureOnRead.java
index 7af57de..d390ca3 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailureOnRead.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailureOnRead.java
@@ -30,7 +30,7 @@ import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.XceiverClientRatis;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
import org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException;
@@ -97,18 +97,19 @@ public class TestContainerStateMachineFailureOnRead {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
conf.setQuietMode(false);
cluster = MiniOzoneCluster.newBuilder(conf)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java
index fba3ba1..e266ff4 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java
@@ -24,7 +24,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
import org.apache.hadoop.hdds.scm.XceiverClientSpi;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
@@ -121,23 +121,24 @@ public class TestContainerStateMachineFailures {
conf.setTimeDuration(OZONE_SCM_PIPELINE_DESTROY_TIMEOUT, 1,
TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 10, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 10, TimeUnit.SECONDS);
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(ratisClientConfig);
+
DatanodeRatisServerConfig ratisServerConfig =
conf.getObject(DatanodeRatisServerConfig.class);
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
conf.setLong(OzoneConfigKeys.DFS_RATIS_SNAPSHOT_THRESHOLD_KEY, 1);
conf.setQuietMode(false);
cluster =
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java
index cac16f1..3dc46b9 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java
@@ -31,7 +31,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
import org.apache.hadoop.hdds.scm.XceiverClientSpi;
@@ -120,20 +120,22 @@ public class TestDeleteWithSlowFollower {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
conf.setTimeDuration(OzoneConfigKeys.OZONE_BLOCK_DELETING_SERVICE_INTERVAL,
1, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
conf.setQuietMode(false);
int numOfDatanodes = 3;
cluster = MiniOzoneCluster.newBuilder(conf)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java
index e35a393..887acbb 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java
@@ -32,7 +32,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.ContainerInfo;
@@ -97,10 +97,13 @@ public class TestFailureHandlingByClient {
chunkSize = (int) OzoneConsts.MB;
blockSize = 4 * chunkSize;
conf.setTimeDuration(OZONE_SCM_STALENODE_INTERVAL, 100, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
conf.setTimeDuration(
OzoneConfigKeys.DFS_RATIS_LEADER_ELECTION_MINIMUM_TIMEOUT_DURATION_KEY,
1, TimeUnit.SECONDS);
@@ -112,14 +115,13 @@ public class TestFailureHandlingByClient {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
conf.setBoolean(
OzoneConfigKeys.OZONE_CLIENT_STREAM_BUFFER_FLUSH_DELAY, false);
conf.setQuietMode(false);
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClientFlushDelay.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClientFlushDelay.java
index 8f145b4..a7bf514 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClientFlushDelay.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClientFlushDelay.java
@@ -25,7 +25,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.ContainerInfo;
@@ -95,10 +95,13 @@ public class TestFailureHandlingByClientFlushDelay {
maxFlushSize = 2 * flushSize;
blockSize = 4 * chunkSize;
conf.setTimeDuration(OZONE_SCM_STALENODE_INTERVAL, 100, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
conf.setTimeDuration(
OzoneConfigKeys.DFS_RATIS_LEADER_ELECTION_MINIMUM_TIMEOUT_DURATION_KEY,
1, TimeUnit.SECONDS);
@@ -110,14 +113,13 @@ public class TestFailureHandlingByClientFlushDelay {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
conf.setQuietMode(false);
conf.setClass(NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY,
StaticMapping.class, DNSToSwitchMapping.class);
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java
index 1ca073d..1329cc3 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java
@@ -23,7 +23,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.container.ContainerID;
import org.apache.hadoop.hdds.scm.container.ContainerInfo;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
@@ -94,18 +94,20 @@ public class TestMultiBlockWritesWithDnFailures {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 30, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 30, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ conf.setFromObject(ratisClientConfig);
+
conf.setTimeDuration(
OzoneConfigKeys.DFS_RATIS_LEADER_ELECTION_MINIMUM_TIMEOUT_DURATION_KEY,
1, TimeUnit.SECONDS);
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestValidateBCSIDOnRestart.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestValidateBCSIDOnRestart.java
index 63e63af..315a1df 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestValidateBCSIDOnRestart.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestValidateBCSIDOnRestart.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
import org.apache.hadoop.ozone.HddsDatanodeService;
import org.apache.hadoop.ozone.MiniOzoneCluster;
@@ -106,24 +106,24 @@ public class TestValidateBCSIDOnRestart {
conf.setTimeDuration(OZONE_SCM_PIPELINE_DESTROY_TIMEOUT, 1,
TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 10, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 10, TimeUnit.SECONDS);
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(ratisClientConfig);
+
DatanodeRatisServerConfig ratisServerConfig =
conf.getObject(DatanodeRatisServerConfig.class);
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY + "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
- conf.setQuietMode(false);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
cluster =
MiniOzoneCluster.newBuilder(conf).setNumDatanodes(2).
setHbInterval(200)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
index 131d289..ed9642a 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
@@ -22,7 +22,7 @@ import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.*;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline;
@@ -100,23 +100,25 @@ public class TestWatchForCommit {
conf.setTimeDuration(OZONE_SCM_PIPELINE_DESTROY_TIMEOUT, 10,
TimeUnit.SECONDS);
conf.setQuietMode(false);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.write.timeout", 10, TimeUnit.SECONDS);
- conf.setTimeDuration(RatisHelper.HDDS_DATANODE_RATIS_PREFIX_KEY
- + ".client.request.watch.timeout", 10, TimeUnit.SECONDS);
+
+ RatisClientConfig ratisClientConfig =
+ conf.getObject(RatisClientConfig.class);
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(ratisClientConfig);
+
DatanodeRatisServerConfig ratisServerConfig =
conf.getObject(DatanodeRatisServerConfig.class);
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
conf.setTimeDuration(OZONE_SCM_STALENODE_INTERVAL, 30, TimeUnit.SECONDS);
cluster = MiniOzoneCluster.newBuilder(conf)
.setNumDatanodes(9)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java
index 154bd55..86b004d 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java
@@ -23,7 +23,7 @@ import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.junit.Assert;
import org.junit.Test;
@@ -44,14 +44,13 @@ public abstract class TestDataValidate {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(10));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 10, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(10));
+ conf.setFromObject(raftClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf)
.setNumDatanodes(5).setTotalPipelineNumLimit(8).build();
cluster.waitForClusterToBeReady();
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java
index 8984b66..1207bab 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java
@@ -24,7 +24,7 @@ import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.junit.AfterClass;
@@ -74,14 +74,13 @@ public class TestFreonWithDatanodeRestart {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf)
.setHbProcessorInterval(1000)
.setHbInterval(1000)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java
index 61ac6af..615444f 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java
@@ -25,7 +25,7 @@ import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
@@ -67,14 +67,13 @@ public class TestFreonWithPipelineDestroy {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf)
.setHbProcessorInterval(1000)
.setHbInterval(1000)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java
index ddb27ce..f05ce07 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java
@@ -25,7 +25,7 @@ import org.apache.hadoop.hdds.client.ReplicationFactor;
import org.apache.hadoop.hdds.client.ReplicationType;
import org.apache.hadoop.hdds.conf.DatanodeRatisServerConfig;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
-import org.apache.hadoop.hdds.ratis.RatisHelper;
+import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
import org.apache.hadoop.ozone.MiniOzoneCluster;
import org.junit.AfterClass;
@@ -57,14 +57,13 @@ public class TestRandomKeyGenerator {
ratisServerConfig.setRequestTimeOut(Duration.ofSeconds(3));
ratisServerConfig.setWatchTimeOut(Duration.ofSeconds(3));
conf.setFromObject(ratisServerConfig);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "rpc.request.timeout",
- 3, TimeUnit.SECONDS);
- conf.setTimeDuration(
- RatisHelper.HDDS_DATANODE_RATIS_CLIENT_PREFIX_KEY+ "." +
- "watch.request.timeout",
- 3, TimeUnit.SECONDS);
+
+ RatisClientConfig.RaftConfig raftClientConfig =
+ conf.getObject(RatisClientConfig.RaftConfig.class);
+ raftClientConfig.setRpcRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ raftClientConfig.setRpcWatchRequestTimeout(TimeUnit.SECONDS.toMillis(3));
+ conf.setFromObject(raftClientConfig);
+
cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(5).build();
cluster.waitForClusterToBeReady();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 24/39: HDDS-3824: OM read requests should make
SCM#refreshPipeline outside BUCKET_LOCK (#1164)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit d4c6f7dde21483acf73a1c1d1e67ca97accfa888
Author: Rakesh Radhakrishnan <ra...@apache.org>
AuthorDate: Fri Jul 17 02:25:07 2020 +0530
HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK (#1164)
(cherry picked from commit 46e7b2f90a5cbac5aa7363e517e870db8d0c6fba)
---
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 164 +++++++++++----------
1 file changed, 89 insertions(+), 75 deletions(-)
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
index 9e12e13..97af784 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
@@ -638,48 +638,53 @@ public class KeyManagerImpl implements KeyManager {
String keyName = args.getKeyName();
metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
bucketName);
+ OmKeyInfo value = null;
try {
String keyBytes = metadataManager.getOzoneKey(
volumeName, bucketName, keyName);
- OmKeyInfo value = metadataManager.getKeyTable().get(keyBytes);
- if (value == null) {
- LOG.debug("volume:{} bucket:{} Key:{} not found",
- volumeName, bucketName, keyName);
- throw new OMException("Key not found",
- KEY_NOT_FOUND);
- }
- if (grpcBlockTokenEnabled) {
- String remoteUser = getRemoteUser().getShortUserName();
- for (OmKeyLocationInfoGroup key : value.getKeyLocationVersions()) {
- key.getLocationList().forEach(k -> {
- k.setToken(secretManager.generateToken(remoteUser,
- k.getBlockID().getContainerBlockID().toString(),
- getAclForUser(remoteUser),
- k.getLength()));
- });
- }
- }
- // Refresh container pipeline info from SCM
- // based on OmKeyArgs.refreshPipeline flag
- if (args.getRefreshPipeline()) {
- refreshPipeline(value);
- }
- if (args.getSortDatanodes()) {
- sortDatanodeInPipeline(value, clientAddress);
- }
- return value;
+ value = metadataManager.getKeyTable().get(keyBytes);
} catch (IOException ex) {
if (ex instanceof OMException) {
throw ex;
}
- LOG.debug("Get key failed for volume:{} bucket:{} key:{}",
- volumeName, bucketName, keyName, ex);
- throw new OMException(ex.getMessage(),
- KEY_NOT_FOUND);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Get key failed for volume:{} bucket:{} key:{}", volumeName,
+ bucketName, keyName, ex);
+ }
+ throw new OMException(ex.getMessage(), KEY_NOT_FOUND);
} finally {
metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
bucketName);
}
+
+ if (value == null) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("volume:{} bucket:{} Key:{} not found", volumeName,
+ bucketName, keyName);
+ }
+ throw new OMException("Key not found", KEY_NOT_FOUND);
+ }
+ if (grpcBlockTokenEnabled) {
+ String remoteUser = getRemoteUser().getShortUserName();
+ for (OmKeyLocationInfoGroup key : value.getKeyLocationVersions()) {
+ key.getLocationList().forEach(k -> {
+ k.setToken(secretManager.generateToken(remoteUser,
+ k.getBlockID().getContainerBlockID().toString(),
+ getAclForUser(remoteUser), k.getLength()));
+ });
+ }
+ }
+
+ // Refresh container pipeline info from SCM
+ // based on OmKeyArgs.refreshPipeline flag
+ // value won't be null as the check is done inside try/catch block.
+ if (args.getRefreshPipeline()) {
+ refreshPipeline(value);
+ }
+ if (args.getSortDatanodes()) {
+ sortDatanodeInPipeline(value, clientAddress);
+ }
+ return value;
}
/**
@@ -1695,6 +1700,18 @@ public class KeyManagerImpl implements KeyManager {
String bucketName = args.getBucketName();
String keyName = args.getKeyName();
+ return getOzoneFileStatus(volumeName, bucketName, keyName,
+ args.getRefreshPipeline(), false, null);
+ }
+
+ private OzoneFileStatus getOzoneFileStatus(String volumeName,
+ String bucketName,
+ String keyName,
+ boolean refreshPipeline,
+ boolean sortDatanodes,
+ String clientAddress)
+ throws IOException {
+ OmKeyInfo fileKeyInfo = null;
metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
bucketName);
try {
@@ -1706,36 +1723,44 @@ public class KeyManagerImpl implements KeyManager {
// Check if the key is a file.
String fileKeyBytes = metadataManager.getOzoneKey(
- volumeName, bucketName, keyName);
- OmKeyInfo fileKeyInfo = metadataManager.getKeyTable().get(fileKeyBytes);
+ volumeName, bucketName, keyName);
+ fileKeyInfo = metadataManager.getKeyTable().get(fileKeyBytes);
+
+ // Check if the key is a directory.
+ if (fileKeyInfo == null) {
+ String dirKey = OzoneFSUtils.addTrailingSlashIfNeeded(keyName);
+ String dirKeyBytes = metadataManager.getOzoneKey(
+ volumeName, bucketName, dirKey);
+ OmKeyInfo dirKeyInfo = metadataManager.getKeyTable().get(dirKeyBytes);
+ if (dirKeyInfo != null) {
+ return new OzoneFileStatus(dirKeyInfo, scmBlockSize, true);
+ }
+ }
+ } finally {
+ metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
+ bucketName);
+
+ // if the key is a file then do refresh pipeline info in OM by asking SCM
if (fileKeyInfo != null) {
- if (args.getRefreshPipeline()) {
+ if (refreshPipeline) {
refreshPipeline(fileKeyInfo);
}
- // this is a file
+ if (sortDatanodes) {
+ sortDatanodeInPipeline(fileKeyInfo, clientAddress);
+ }
return new OzoneFileStatus(fileKeyInfo, scmBlockSize, false);
}
+ }
- String dirKey = OzoneFSUtils.addTrailingSlashIfNeeded(keyName);
- String dirKeyBytes = metadataManager.getOzoneKey(
- volumeName, bucketName, dirKey);
- OmKeyInfo dirKeyInfo = metadataManager.getKeyTable().get(dirKeyBytes);
- if (dirKeyInfo != null) {
- return new OzoneFileStatus(dirKeyInfo, scmBlockSize, true);
- }
-
- if (LOG.isDebugEnabled()) {
- LOG.debug("Unable to get file status for the key: volume: {}, bucket:" +
- " {}, key: {}, with error: No such file exists.", volumeName,
- bucketName, keyName);
- }
- throw new OMException("Unable to get file status: volume: " +
- volumeName + " bucket: " + bucketName + " key: " + keyName,
- FILE_NOT_FOUND);
- } finally {
- metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
- bucketName);
+ // Key is not found, throws exception
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Unable to get file status for the key: volume: {}, bucket:" +
+ " {}, key: {}, with error: No such file exists.",
+ volumeName, bucketName, keyName);
}
+ throw new OMException("Unable to get file status: volume: " +
+ volumeName + " bucket: " + bucketName + " key: " + keyName,
+ FILE_NOT_FOUND);
}
/**
@@ -1877,26 +1902,13 @@ public class KeyManagerImpl implements KeyManager {
String volumeName = args.getVolumeName();
String bucketName = args.getBucketName();
String keyName = args.getKeyName();
-
- metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
- bucketName);
- try {
- OzoneFileStatus fileStatus = getFileStatus(args);
- if (fileStatus.isFile()) {
- if (args.getRefreshPipeline()) {
- refreshPipeline(fileStatus.getKeyInfo());
- }
- if (args.getSortDatanodes()) {
- sortDatanodeInPipeline(fileStatus.getKeyInfo(), clientAddress);
- }
- return fileStatus.getKeyInfo();
- }
+ OzoneFileStatus fileStatus = getOzoneFileStatus(volumeName, bucketName,
+ keyName, args.getRefreshPipeline(), args.getSortDatanodes(),
+ clientAddress);
//if key is not of type file or if key is not found we throw an exception
- } finally {
- metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
- bucketName);
+ if (fileStatus.isFile()) {
+ return fileStatus.getKeyInfo();
}
-
throw new OMException("Can not write to directory: " + keyName,
ResultCodes.NOT_A_FILE);
}
@@ -2067,9 +2079,6 @@ public class KeyManagerImpl implements KeyManager {
for (Map.Entry<String, OzoneFileStatus> entry : cacheKeyMap.entrySet()) {
// No need to check if a key is deleted or not here, this is handled
// when adding entries to cacheKeyMap from DB.
- if (args.getRefreshPipeline()) {
- refreshPipeline(entry.getValue().getKeyInfo());
- }
fileStatusList.add(entry.getValue());
countEntries++;
if (countEntries >= numEntries) {
@@ -2083,6 +2092,11 @@ public class KeyManagerImpl implements KeyManager {
metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
bucketName);
}
+ if (args.getRefreshPipeline()) {
+ for(OzoneFileStatus fileStatus : fileStatusList){
+ refreshPipeline(fileStatus.getKeyInfo());
+ }
+ }
return fileStatusList;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 30/39: HDDS-3955. Unable to list intermediate paths
on keys created using S3G. (#1196)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit ba8966d0b109d67b8ac2c9b0604da02774482808
Author: Bharat Viswanadham <bh...@apache.org>
AuthorDate: Sat Jul 18 10:15:45 2020 -0700
HDDS-3955. Unable to list intermediate paths on keys created using S3G. (#1196)
(cherry picked from commit 715aed2d158d2c6708af8b6a9b8270766103ee52)
---
.../common/src/main/resources/ozone-default.xml | 15 ++
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 7 +
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 132 +++++++++++++++
.../org/apache/hadoop/ozone/om/OzoneManager.java | 8 +
.../hadoop/ozone/om/request/OMClientRequest.java | 76 +++++++++
.../ozone/om/request/file/OMFileRequest.java | 2 +-
.../om/request/key/OMAllocateBlockRequest.java | 6 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 68 +++++++-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 4 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 17 +-
.../S3InitiateMultipartUploadRequest.java | 14 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 9 +-
.../S3MultipartUploadCompleteRequest.java | 5 +-
.../ozone/om/request/TestNormalizePaths.java | 109 ++++++++++++
.../om/request/key/TestOMKeyCreateRequest.java | 182 ++++++++++++++++++++-
.../TestS3InitiateMultipartUploadRequest.java | 2 +-
.../s3/multipart/TestS3MultipartRequest.java | 5 +-
.../TestS3MultipartUploadCommitPartRequest.java | 2 +-
20 files changed, 643 insertions(+), 29 deletions(-)
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index d8fc591..b474ac3 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -2478,4 +2478,19 @@
rules in Amazon S3's object key naming guide.
</description>
</property>
+
+ <property>
+ <name>ozone.om.enable.filesystem.paths</name>
+ <tag>OZONE, OM</tag>
+ <value>false</value>
+ <description>If true, key names will be interpreted as file system paths.
+ "/" will be treated as a special character and paths will be normalized
+ and must follow Unix filesystem path naming conventions. This flag will
+ be helpful when objects created by S3G need to be accessed using OFS/O3Fs.
+ If false, it will fallback to default behavior of Key/MPU create
+ requests where key paths are not normalized and any intermediate
+ directories will not be created or any file checks happens to check
+ filesystem semantics.
+ </description>
+ </property>
</configuration>
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
index 4f512a5..f16679a 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
@@ -239,4 +239,11 @@ public final class OMConfigKeys {
"ozone.om.keyname.character.check.enabled";
public static final boolean OZONE_OM_KEYNAME_CHARACTER_CHECK_ENABLED_DEFAULT =
false;
+
+ // This config needs to be enabled, when S3G created objects used via
+ // FileSystem API.
+ public static final String OZONE_OM_ENABLE_FILESYSTEM_PATHS =
+ "ozone.om.enable.filesystem.paths";
+ public static final boolean OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT =
+ false;
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSWithObjectStoreCreate.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSWithObjectStoreCreate.java
new file mode 100644
index 0000000..b872a3d
--- /dev/null
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSWithObjectStoreCreate.java
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.ozone;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.TestDataUtil;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+import java.net.URI;
+import java.util.Arrays;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_SCHEME;
+
+/**
+ * Class tests create with object store and getFileStatus.
+ */
+public class TestOzoneFSWithObjectStoreCreate {
+
+ @Rule
+ public Timeout timeout = new Timeout(300000);
+
+ private String rootPath;
+
+ private MiniOzoneCluster cluster = null;
+
+ private OzoneFileSystem o3fs;
+
+ private String volumeName;
+
+ private String bucketName;
+
+
+ @Before
+ public void init() throws Exception {
+ volumeName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ bucketName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+
+ OzoneConfiguration conf = new OzoneConfiguration();
+
+ conf.setBoolean(OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+ cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(3)
+ .build();
+ cluster.waitForClusterToBeReady();
+
+ // create a volume and a bucket to be used by OzoneFileSystem
+ OzoneBucket bucket =
+ TestDataUtil.createVolumeAndBucket(cluster, volumeName, bucketName);
+
+ rootPath = String.format("%s://%s.%s/", OZONE_URI_SCHEME, bucketName,
+ volumeName);
+ o3fs = (OzoneFileSystem) FileSystem.get(new URI(rootPath), conf);
+ }
+
+
+ @Test
+ public void test() throws Exception {
+
+ OzoneVolume ozoneVolume =
+ cluster.getRpcClient().getObjectStore().getVolume(volumeName);
+
+ OzoneBucket ozoneBucket = ozoneVolume.getBucket(bucketName);
+
+ String key1 = "///dir1/dir2/file1";
+ String key2 = "///dir1/dir2/file2";
+ int length = 10;
+ OzoneOutputStream ozoneOutputStream = ozoneBucket.createKey(key1, length);
+ byte[] b = new byte[10];
+ Arrays.fill(b, (byte)96);
+ ozoneOutputStream.write(b);
+ ozoneOutputStream.close();
+
+ ozoneOutputStream = ozoneBucket.createKey(key2, length);
+ ozoneOutputStream.write(b);
+ ozoneOutputStream.close();
+
+ // Adding "/" here otherwise Path will be considered as relative path and
+ // workingDir will be added.
+ key1 = "///dir1/dir2/file1";
+ Path p = new Path(key1);
+ Assert.assertTrue(o3fs.getFileStatus(p).isFile());
+
+ p = p.getParent();
+ checkAncestors(p);
+
+
+ key2 = "///dir1/dir2/file2";
+ p = new Path(key2);
+ Assert.assertTrue(o3fs.getFileStatus(p).isFile());
+ checkAncestors(p);
+
+ }
+
+ private void checkAncestors(Path p) throws Exception {
+ p = p.getParent();
+ while(p.getParent() != null) {
+ FileStatus fileStatus = o3fs.getFileStatus(p);
+ Assert.assertTrue(fileStatus.isDirectory());
+ p = p.getParent();
+ }
+ }
+
+}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 8a49fa7..43ae998 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@ -200,6 +200,8 @@ import static org.apache.hadoop.ozone.OzoneConsts.OM_METRICS_FILE;
import static org.apache.hadoop.ozone.OzoneConsts.OM_METRICS_TEMP_FILE;
import static org.apache.hadoop.ozone.OzoneConsts.RPC_PORT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ADDRESS_KEY;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_HANDLER_COUNT_DEFAULT;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_HANDLER_COUNT_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_KERBEROS_KEYTAB_FILE_KEY;
@@ -3494,4 +3496,10 @@ public final class OzoneManager extends ServiceRuntimeInfoImpl
void setExitManagerForTesting(ExitManager exitManagerForTesting) {
this.exitManager = exitManagerForTesting;
}
+
+
+ public boolean getEnableFileSystemPaths() {
+ return configuration.getBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS,
+ OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
+ }
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
index 4ced9fd..0fa9ca1 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.ozone.om.request;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.ipc.ProtobufRpcEngine;
import org.apache.hadoop.ozone.OzoneConsts;
@@ -38,13 +39,18 @@ import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRespo
import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
import org.apache.hadoop.ozone.security.acl.OzoneObj;
import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.net.InetAddress;
+import java.nio.file.Paths;
import java.util.LinkedHashMap;
import java.util.Map;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_KEY_NAME;
/**
* OMClientRequest provides methods which every write OM request should
@@ -52,6 +58,8 @@ import java.util.Map;
*/
public abstract class OMClientRequest implements RequestAuditor {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMClientRequest.class);
private OMRequest omRequest;
/**
@@ -265,4 +273,72 @@ public abstract class OMClientRequest implements RequestAuditor {
auditMap.put(OzoneConsts.VOLUME, volume);
return auditMap;
}
+
+
+ public static String validateAndNormalizeKey(boolean enableFileSystemPaths,
+ String keyName) throws OMException {
+ if (enableFileSystemPaths) {
+ return validateAndNormalizeKey(keyName);
+ } else {
+ return keyName;
+ }
+ }
+
+ @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+ public static String validateAndNormalizeKey(String keyName)
+ throws OMException {
+ String normalizedKeyName;
+ if (keyName.startsWith(OM_KEY_PREFIX)) {
+ normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+ } else {
+ normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+ .normalize().getPath();
+ }
+ if (!keyName.equals(normalizedKeyName)) {
+ LOG.debug("Normalized key {} to {} ", keyName,
+ normalizedKeyName.substring(1));
+ }
+ return isValidKeyPath(normalizedKeyName.substring(1));
+ }
+
+ /**
+ * Whether the pathname is valid. Check key names which contain a
+ * ":", ".", "..", "//", "". If it has any of these characters throws
+ * OMException, else return the path.
+ */
+ private static String isValidKeyPath(String path) throws OMException {
+ boolean isValid = true;
+
+ // If keyName is empty string throw error.
+ if (path.length() == 0) {
+ throw new OMException("Invalid KeyPath, empty keyName" + path,
+ INVALID_KEY_NAME);
+ } else if(path.startsWith("/")) {
+ isValid = false;
+ } else {
+ // Check for ".." "." ":" "/"
+ String[] components = StringUtils.split(path, '/');
+ for (int i = 0; i < components.length; i++) {
+ String element = components[i];
+ if (element.equals(".") ||
+ (element.contains(":")) ||
+ (element.contains("/") || element.equals(".."))) {
+ isValid = false;
+ break;
+ }
+
+ // The string may end with a /, but not have
+ // "//" in the middle.
+ if (element.isEmpty() && i != components.length - 1) {
+ isValid = false;
+ }
+ }
+ }
+
+ if (isValid) {
+ return path;
+ } else {
+ throw new OMException("Invalid KeyPath " + path, INVALID_KEY_NAME);
+ }
+ }
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
index 3367ec7..21ffff8 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
@@ -197,7 +197,7 @@ public final class OMFileRequest {
/**
* Return codes used by verifyFilesInPath method.
*/
- enum OMDirectoryResult {
+ public enum OMDirectoryResult {
// In below examples path is assumed as "a/b/c" in volume volume1 and
// bucket b1.
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
index 9e82888..94d700f 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
@@ -110,9 +110,11 @@ public class OMAllocateBlockRequest extends OMKeyRequest {
ozoneManager.getPreallocateBlocksMax(),
ozoneManager.isGrpcBlockTokenEnabled(), ozoneManager.getOMNodeId());
- // Set modification time
+ // Set modification time and normalize key if required.
KeyArgs.Builder newKeyArgs = keyArgs.toBuilder()
- .setModificationTime(Time.now());
+ .setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(), keyArgs.getKeyName()));
AllocateBlockRequest.Builder newAllocatedBlockRequest =
AllocateBlockRequest.newBuilder()
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
index eb3769b..dccb93b 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
@@ -91,7 +91,9 @@ public class OMKeyCommitRequest extends OMKeyRequest {
}
KeyArgs.Builder newKeyArgs =
- keyArgs.toBuilder().setModificationTime(Time.now());
+ keyArgs.toBuilder().setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(), keyArgs.getKeyName()));
return getOmRequest().toBuilder()
.setCommitKeyRequest(commitKeyRequest.toBuilder()
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
index f7f08dc..8927c19 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.ozone.om.request.key;
import java.io.IOException;
+import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -27,7 +28,11 @@ import java.util.stream.Collectors;
import com.google.common.base.Optional;
import com.google.common.base.Preconditions;
import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.OzoneAcl;
import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.request.file.OMDirectoryCreateRequest;
+import org.apache.hadoop.ozone.om.request.file.OMFileRequest;
import org.apache.hadoop.ozone.om.request.util.OmResponseUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -63,7 +68,10 @@ import org.apache.hadoop.ozone.security.acl.OzoneObj;
import org.apache.hadoop.util.Time;
import org.apache.hadoop.hdds.utils.UniqueId;
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.NOT_A_FILE;
import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+import static org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.DIRECTORY_EXISTS;
+import static org.apache.hadoop.ozone.om.request.file.OMFileRequest.OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
/**
* Handles CreateKey request.
@@ -91,6 +99,20 @@ public class OMKeyCreateRequest extends OMKeyRequest {
if(checkKeyNameEnabled){
OmUtils.validateKeyName(keyArgs.getKeyName());
}
+
+ String keyPath = keyArgs.getKeyName();
+ if (ozoneManager.getEnableFileSystemPaths()) {
+ // If enabled, disallow keys with trailing /. As in fs semantics
+ // directories end with trailing /.
+ keyPath = validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(), keyPath);
+ if (keyPath.endsWith("/")) {
+ throw new OMException("Invalid KeyPath, key names with trailing / " +
+ "are not allowed." + keyPath,
+ OMException.ResultCodes.INVALID_KEY_NAME);
+ }
+ }
+
// We cannot allocate block for multipart upload part when
// createMultipartKey is called, as we will not know type and factor with
// which initiateMultipartUpload has started for this key. When
@@ -131,7 +153,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
// As for a client for the first time this can be executed on any OM,
// till leader is identified.
- List< OmKeyLocationInfo > omKeyLocationInfoList =
+ List<OmKeyLocationInfo> omKeyLocationInfoList =
allocateBlock(ozoneManager.getScmClient(),
ozoneManager.getBlockTokenSecretManager(), type, factor,
new ExcludeList(), requestedSize, scmBlockSize,
@@ -149,7 +171,10 @@ public class OMKeyCreateRequest extends OMKeyRequest {
newKeyArgs = keyArgs.toBuilder().setModificationTime(Time.now());
}
+ newKeyArgs.setKeyName(keyPath);
+
generateRequiredEncryptionInfo(keyArgs, newKeyArgs, ozoneManager);
+
newCreateKeyRequest =
createKeyRequest.toBuilder().setKeyArgs(newKeyArgs)
.setClientID(UniqueId.next());
@@ -160,6 +185,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
}
@Override
+ @SuppressWarnings("methodlength")
public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
long trxnLogIndex, OzoneManagerDoubleBufferHelper omDoubleBufferHelper) {
CreateKeyRequest createKeyRequest = getOmRequest().getCreateKeyRequest();
@@ -184,6 +210,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
getOmRequest());
IOException exception = null;
Result result = null;
+ List<OmKeyInfo> missingParentInfos = null;
try {
keyArgs = resolveBucketLink(ozoneManager, keyArgs, auditMap);
volumeName = keyArgs.getVolumeName();
@@ -209,8 +236,41 @@ public class OMKeyCreateRequest extends OMKeyRequest {
OmBucketInfo bucketInfo = omMetadataManager.getBucketTable().get(
omMetadataManager.getBucketKey(volumeName, bucketName));
+ // If FILE_EXISTS we just override like how we used to do for Key Create.
+ List< OzoneAcl > inheritAcls;
+ if (ozoneManager.getEnableFileSystemPaths()) {
+ OMFileRequest.OMPathInfo pathInfo =
+ OMFileRequest.verifyFilesInPath(omMetadataManager, volumeName,
+ bucketName, keyName, Paths.get(keyName));
+ OMFileRequest.OMDirectoryResult omDirectoryResult =
+ pathInfo.getDirectoryResult();
+ inheritAcls = pathInfo.getAcls();
+
+ // Check if a file or directory exists with same key name.
+ if (omDirectoryResult == DIRECTORY_EXISTS) {
+ throw new OMException("Cannot write to " +
+ "directory. createIntermediateDirs behavior is enabled and " +
+ "hence / has special interpretation: " + keyName, NOT_A_FILE);
+ } else
+ if (omDirectoryResult == FILE_EXISTS_IN_GIVENPATH) {
+ throw new OMException("Can not create file: " + keyName +
+ " as there is already file in the given path", NOT_A_FILE);
+ }
+
+ missingParentInfos = OMDirectoryCreateRequest
+ .getAllParentInfo(ozoneManager, keyArgs,
+ pathInfo.getMissingParents(), inheritAcls, trxnLogIndex);
+
+ // Add cache entries for the prefix directories.
+ // Skip adding for the file key itself, until Key Commit.
+ OMFileRequest.addKeyTableCacheEntries(omMetadataManager, volumeName,
+ bucketName, Optional.absent(), Optional.of(missingParentInfos),
+ trxnLogIndex);
+
+ }
+
omKeyInfo = prepareKeyInfo(omMetadataManager, keyArgs, dbKeyInfo,
- keyArgs.getDataSize(), locations, getFileEncryptionInfo(keyArgs),
+ keyArgs.getDataSize(), locations, getFileEncryptionInfo(keyArgs),
ozoneManager.getPrefixManager(), bucketInfo, trxnLogIndex,
ozoneManager.isRatisEnabled());
@@ -238,7 +298,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
.setOpenVersion(openVersion).build())
.setCmdType(Type.CreateKey);
omClientResponse = new OMKeyCreateResponse(omResponse.build(),
- omKeyInfo, null, clientID);
+ omKeyInfo, missingParentInfos, clientID);
result = Result.SUCCESS;
} catch (IOException ex) {
@@ -269,7 +329,7 @@ public class OMKeyCreateRequest extends OMKeyRequest {
break;
case FAILURE:
LOG.error("Key creation failed. Volume:{}, Bucket:{}, Key{}. " +
- "Exception:{}", volumeName, bucketName, keyName, exception);
+ "Exception:{}", volumeName, bucketName, keyName, exception);
break;
default:
LOG.error("Unrecognized Result for OMKeyCreateRequest: {}",
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
index 8b75417..4d8562c 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
@@ -75,7 +75,9 @@ public class OMKeyDeleteRequest extends OMKeyRequest {
OzoneManagerProtocolProtos.KeyArgs keyArgs = deleteKeyRequest.getKeyArgs();
OzoneManagerProtocolProtos.KeyArgs.Builder newKeyArgs =
- keyArgs.toBuilder().setModificationTime(Time.now());
+ keyArgs.toBuilder().setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(), keyArgs.getKeyName()));
return getOmRequest().toBuilder()
.setDeleteKeyRequest(deleteKeyRequest.toBuilder()
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
index 91db347..e6e9839 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
@@ -85,13 +85,22 @@ public class OMKeyRenameRequest extends OMKeyRequest {
OmUtils.validateKeyName(renameKeyRequest.getToKeyName());
}
- // Set modification time.
- KeyArgs.Builder newKeyArgs = renameKeyRequest.getKeyArgs().toBuilder()
- .setModificationTime(Time.now());
+ KeyArgs renameKeyArgs = renameKeyRequest.getKeyArgs();
+
+ // Set modification time and normalize key if needed.
+ KeyArgs.Builder newKeyArgs = renameKeyArgs.toBuilder()
+ .setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(),
+ renameKeyArgs.getKeyName()));
return getOmRequest().toBuilder()
.setRenameKeyRequest(renameKeyRequest.toBuilder()
- .setKeyArgs(newKeyArgs)).setUserInfo(getUserInfo()).build();
+ .setKeyArgs(newKeyArgs)
+ .setToKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(),
+ renameKeyRequest.getToKeyName())))
+ .setUserInfo(getUserInfo()).build();
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
index aa96ba9..f7951a2 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3InitiateMultipartUploadRequest.java
@@ -33,7 +33,7 @@ import org.apache.hadoop.ozone.om.request.key.OMKeyRequest;
import org.apache.hadoop.ozone.om.request.util.OmResponseUtil;
import org.apache.hadoop.ozone.om.response.OMClientResponse;
import org.apache.hadoop.ozone.om.response.s3.multipart.S3InitiateMultipartUploadResponse;
-import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.MultipartInfoInitiateRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.MultipartInfoInitiateResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRequest;
@@ -67,15 +67,17 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
}
@Override
- public OMRequest preExecute(OzoneManager ozoneManager) {
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
MultipartInfoInitiateRequest multipartInfoInitiateRequest =
getOmRequest().getInitiateMultiPartUploadRequest();
Preconditions.checkNotNull(multipartInfoInitiateRequest);
- OzoneManagerProtocolProtos.KeyArgs.Builder newKeyArgs =
- multipartInfoInitiateRequest.getKeyArgs().toBuilder()
+ KeyArgs keyArgs = multipartInfoInitiateRequest.getKeyArgs();
+ KeyArgs.Builder newKeyArgs = keyArgs.toBuilder()
.setMultipartUploadID(UUID.randomUUID().toString() + "-" +
- UniqueId.next()).setModificationTime(Time.now());
+ UniqueId.next()).setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(), keyArgs.getKeyName()));
return getOmRequest().toBuilder()
.setUserInfo(getUserInfo())
@@ -92,7 +94,7 @@ public class S3InitiateMultipartUploadRequest extends OMKeyRequest {
MultipartInfoInitiateRequest multipartInfoInitiateRequest =
getOmRequest().getInitiateMultiPartUploadRequest();
- OzoneManagerProtocolProtos.KeyArgs keyArgs =
+ KeyArgs keyArgs =
multipartInfoInitiateRequest.getKeyArgs();
Preconditions.checkNotNull(keyArgs.getMultipartUploadID());
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
index 0726fe4..c0ef8b3 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java
@@ -73,7 +73,10 @@ public class S3MultipartUploadAbortRequest extends OMKeyRequest {
return getOmRequest().toBuilder().setAbortMultiPartUploadRequest(
getOmRequest().getAbortMultiPartUploadRequest().toBuilder()
- .setKeyArgs(keyArgs.toBuilder().setModificationTime(Time.now())))
+ .setKeyArgs(keyArgs.toBuilder().setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(),
+ keyArgs.getKeyName()))))
.setUserInfo(getUserInfo()).build();
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
index 283a22d..1e29d5f 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
@@ -70,14 +70,17 @@ public class S3MultipartUploadCommitPartRequest extends OMKeyRequest {
}
@Override
- public OMRequest preExecute(OzoneManager ozoneManager) {
+ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
MultipartCommitUploadPartRequest multipartCommitUploadPartRequest =
getOmRequest().getCommitMultiPartUploadRequest();
+ KeyArgs keyArgs = multipartCommitUploadPartRequest.getKeyArgs();
return getOmRequest().toBuilder().setCommitMultiPartUploadRequest(
multipartCommitUploadPartRequest.toBuilder()
- .setKeyArgs(multipartCommitUploadPartRequest.getKeyArgs()
- .toBuilder().setModificationTime(Time.now())))
+ .setKeyArgs(keyArgs.toBuilder().setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(),
+ keyArgs.getKeyName()))))
.setUserInfo(getUserInfo()).build();
}
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
index a9aefa0..83cc28b 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java
@@ -81,7 +81,10 @@ public class S3MultipartUploadCompleteRequest extends OMKeyRequest {
return getOmRequest().toBuilder()
.setCompleteMultiPartUploadRequest(multipartUploadCompleteRequest
.toBuilder().setKeyArgs(keyArgs.toBuilder()
- .setModificationTime(Time.now())))
+ .setModificationTime(Time.now())
+ .setKeyName(validateAndNormalizeKey(
+ ozoneManager.getEnableFileSystemPaths(),
+ keyArgs.getKeyName()))))
.setUserInfo(getUserInfo()).build();
}
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
new file mode 100644
index 0000000..6137444
--- /dev/null
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.om.request;
+
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import static org.apache.hadoop.ozone.om.request.OMClientRequest.validateAndNormalizeKey;
+import static org.junit.Assert.fail;
+
+/**
+ * Class to test normalize paths.
+ */
+public class TestNormalizePaths {
+
+ @Rule
+ public ExpectedException exceptionRule = ExpectedException.none();
+
+ @Test
+ public void testNormalizePathsEnabled() throws Exception {
+
+ Assert.assertEquals("a/b/c/d",
+ validateAndNormalizeKey(true, "a/b/c/d"));
+ Assert.assertEquals("a/b/c/d",
+ validateAndNormalizeKey(true, "/a/b/c/d"));
+ Assert.assertEquals("a/b/c/d",
+ validateAndNormalizeKey(true, "////a/b/c/d"));
+ Assert.assertEquals("a/b/c/d",
+ validateAndNormalizeKey(true, "////a/b/////c/d"));
+ Assert.assertEquals("a/b/c/...../d",
+ validateAndNormalizeKey(true, "////a/b/////c/...../d"));
+ Assert.assertEquals("a/b/d",
+ validateAndNormalizeKey(true, "/a/b/c/../d"));
+ Assert.assertEquals("a",
+ validateAndNormalizeKey(true, "a"));
+ Assert.assertEquals("a/b",
+ validateAndNormalizeKey(true, "/a/./b"));
+ Assert.assertEquals("a/b",
+ validateAndNormalizeKey(true, ".//a/./b"));
+ Assert.assertEquals("a/",
+ validateAndNormalizeKey(true, "/a/."));
+ Assert.assertEquals("b/c",
+ validateAndNormalizeKey(true, "//./b/c/"));
+ Assert.assertEquals("a/b/c/d",
+ validateAndNormalizeKey(true, "a/b/c/d/"));
+ Assert.assertEquals("a/b/c/...../d",
+ validateAndNormalizeKey(true, "////a/b/////c/...../d/"));
+ }
+
+ @Test
+ public void testNormalizeKeyInvalidPaths() throws OMException {
+ checkInvalidPath("/a/b/c/../../../../../d");
+ checkInvalidPath("../a/b/c/");
+ checkInvalidPath("/../..a/b/c/");
+ checkInvalidPath("//");
+ checkInvalidPath("/////");
+ checkInvalidPath("");
+ checkInvalidPath("/");
+ checkInvalidPath("/:/:");
+ }
+
+ private void checkInvalidPath(String keyName) {
+ try {
+ validateAndNormalizeKey(true, keyName);
+ fail("checkInvalidPath failed for path " + keyName);
+ } catch (OMException ex) {
+ Assert.assertTrue(ex.getMessage().contains("Invalid KeyPath"));
+ }
+ }
+
+
+
+ @Test
+ public void testNormalizePathsDisable() throws OMException {
+
+ Assert.assertEquals("/a/b/c/d",
+ validateAndNormalizeKey(false, "/a/b/c/d"));
+ Assert.assertEquals("////a/b/c/d",
+ validateAndNormalizeKey(false, "////a/b/c/d"));
+ Assert.assertEquals("////a/b/////c/d",
+ validateAndNormalizeKey(false, "////a/b/////c/d"));
+ Assert.assertEquals("////a/b/////c/...../d",
+ validateAndNormalizeKey(false, "////a/b/////c/...../d"));
+ Assert.assertEquals("/a/b/c/../d",
+ validateAndNormalizeKey(false, "/a/b/c/../d"));
+ Assert.assertEquals("/a/b/c/../../d",
+ validateAndNormalizeKey(false, "/a/b/c/../../d"));
+ }
+}
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
index b26505b..2b8ffce 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
@@ -18,10 +18,15 @@
package org.apache.hadoop.ozone.om.request.key;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
import java.util.List;
import java.util.UUID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
import org.junit.Assert;
import org.junit.Test;
@@ -37,7 +42,14 @@ import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
.OMRequest;
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.THREE;
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType.RATIS;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS;
import static org.apache.hadoop.ozone.om.request.TestOMRequestUtils.addVolumeAndBucketToDB;
+import static org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Status.NOT_A_FILE;
+import static org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Status.OK;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.when;
/**
* Tests OMCreateKeyRequest class.
@@ -82,7 +94,7 @@ public class TestOMKeyCreateRequest extends TestOMKeyRequest {
omKeyCreateRequest.validateAndUpdateCache(ozoneManager, 100L,
ozoneManagerDoubleBufferHelper);
- Assert.assertEquals(OzoneManagerProtocolProtos.Status.OK,
+ Assert.assertEquals(OK,
omKeyCreateResponse.getOMResponse().getStatus());
// Check open table whether key is added or not.
@@ -310,6 +322,11 @@ public class TestOMKeyCreateRequest extends TestOMKeyRequest {
@SuppressWarnings("parameterNumber")
private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber) {
+ return createKeyRequest(isMultipartKey, partNumber, keyName);
+ }
+
+ private OMRequest createKeyRequest(boolean isMultipartKey, int partNumber,
+ String keyName) {
KeyArgs.Builder keyArgs = KeyArgs.newBuilder()
.setVolumeName(volumeName).setBucketName(bucketName)
@@ -327,7 +344,170 @@ public class TestOMKeyCreateRequest extends TestOMKeyRequest {
.setCmdType(OzoneManagerProtocolProtos.Type.CreateKey)
.setClientId(UUID.randomUUID().toString())
.setCreateKeyRequest(createKeyRequest).build();
+ }
+
+ @Test
+ public void testKeyCreateWithFileSystemPathsEnabled() throws Exception {
+
+ OzoneConfiguration configuration = new OzoneConfiguration();
+ configuration.setBoolean(OZONE_OM_ENABLE_FILESYSTEM_PATHS, true);
+ when(ozoneManager.getConfiguration()).thenReturn(configuration);
+ when(ozoneManager.getEnableFileSystemPaths()).thenReturn(true);
+
+ // Add volume and bucket entries to DB.
+ addVolumeAndBucketToDB(volumeName, bucketName,
+ omMetadataManager);
+
+
+ keyName = "dir1/dir2/dir3/file1";
+ createAndCheck(keyName);
+
+ // Key with leading '/'.
+ String keyName = "/a/b/c/file1";
+ createAndCheck(keyName);
+
+ // Commit openKey entry.
+ TestOMRequestUtils.addKeyToTable(false, volumeName, bucketName,
+ keyName.substring(1), 0L, RATIS, THREE, omMetadataManager);
+
+ // Now create another file in same dir path.
+ keyName = "/a/b/c/file2";
+ createAndCheck(keyName);
+
+ // Create key with multiple /'s
+ // converted to a/b/c/file5
+ keyName = "///a/b///c///file5";
+ createAndCheck(keyName);
+
+ // converted to a/b/c/.../file3
+ keyName = "///a/b///c//.../file3";
+ createAndCheck(keyName);
+
+ // converted to r1/r2
+ keyName = "././r1/r2/";
+ createAndCheck(keyName);
+
+ // converted to ..d1/d2/d3
+ keyName = "..d1/d2/d3/";
+ createAndCheck(keyName);
+
+ // Create a file, where a file already exists in the path.
+ // Now try with a file exists in path. Should fail.
+ keyName = "/a/b/c/file1/file3";
+ checkNotAFile(keyName);
+
+ // Empty keyName.
+ keyName = "";
+ checkNotAValidPath(keyName);
+
+ // Key name ends with /
+ keyName = "/a/./";
+ checkNotAValidPath(keyName);
+
+ keyName = "/////";
+ checkNotAValidPath(keyName);
+
+ keyName = "../../b/c";
+ checkNotAValidPath(keyName);
+
+ keyName = "../../b/c/";
+ checkNotAValidPath(keyName);
+
+ keyName = "../../b:/c/";
+ checkNotAValidPath(keyName);
+
+ keyName = ":/c/";
+ checkNotAValidPath(keyName);
+
+ keyName = "";
+ checkNotAValidPath(keyName);
+
+ keyName = "../a/b";
+ checkNotAValidPath(keyName);
+
+ keyName = "/../a/b";
+ checkNotAValidPath(keyName);
+
+ }
+
+ private void checkNotAValidPath(String keyName) {
+ OMRequest omRequest = createKeyRequest(false, 0, keyName);
+ OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+ try {
+ omKeyCreateRequest.preExecute(ozoneManager);
+ fail("checkNotAValidPath failed for path" + keyName);
+ } catch (IOException ex) {
+ Assert.assertTrue(ex instanceof OMException);
+ OMException omException = (OMException) ex;
+ Assert.assertEquals(OMException.ResultCodes.INVALID_KEY_NAME,
+ omException.getResult());
+ }
+
+
+ }
+ private void checkNotAFile(String keyName) throws Exception {
+ OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+ OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+ omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+ omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+ OMClientResponse omClientResponse =
+ omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+ 101L, ozoneManagerDoubleBufferHelper);
+
+ Assert.assertEquals(NOT_A_FILE,
+ omClientResponse.getOMResponse().getStatus());
+ }
+
+
+ private void createAndCheck(String keyName) throws Exception {
+ OMRequest omRequest = createKeyRequest(false, 0, keyName);
+
+ OMKeyCreateRequest omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+ omRequest = omKeyCreateRequest.preExecute(ozoneManager);
+
+ omKeyCreateRequest = new OMKeyCreateRequest(omRequest);
+
+ OMClientResponse omClientResponse =
+ omKeyCreateRequest.validateAndUpdateCache(ozoneManager,
+ 101L, ozoneManagerDoubleBufferHelper);
+
+ Assert.assertEquals(OK, omClientResponse.getOMResponse().getStatus());
+
+ checkCreatedPaths(omKeyCreateRequest, omRequest, keyName);
+ }
+
+ private void checkCreatedPaths(OMKeyCreateRequest omKeyCreateRequest,
+ OMRequest omRequest, String keyName) throws Exception {
+ keyName = omKeyCreateRequest.validateAndNormalizeKey(true, keyName);
+ // Check intermediate directories created or not.
+ Path keyPath = Paths.get(keyName);
+ checkIntermediatePaths(keyPath);
+
+ // Check open key entry
+ String openKey = omMetadataManager.getOpenKey(volumeName, bucketName,
+ keyName, omRequest.getCreateKeyRequest().getClientID());
+ OmKeyInfo omKeyInfo = omMetadataManager.getOpenKeyTable().get(openKey);
+ Assert.assertNotNull(omKeyInfo);
+ }
+
+
+
+ private void checkIntermediatePaths(Path keyPath) throws Exception {
+ // Check intermediate paths are created
+ keyPath = keyPath.getParent();
+ while(keyPath != null) {
+ Assert.assertNotNull(omMetadataManager.getKeyTable().get(
+ omMetadataManager.getOzoneDirKey(volumeName, bucketName,
+ keyPath.toString())));
+ keyPath = keyPath.getParent();
+ }
}
}
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java
index 1d78560..5633c72 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3InitiateMultipartUploadRequest.java
@@ -36,7 +36,7 @@ public class TestS3InitiateMultipartUploadRequest
extends TestS3MultipartRequest {
@Test
- public void testPreExecute() {
+ public void testPreExecute() throws Exception {
doPreExecuteInitiateMPU(UUID.randomUUID().toString(),
UUID.randomUUID().toString(), UUID.randomUUID().toString());
}
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
index 0271a7a..f0f040f 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartRequest.java
@@ -107,7 +107,7 @@ public class TestS3MultipartRequest {
* @return OMRequest - returned from preExecute.
*/
protected OMRequest doPreExecuteInitiateMPU(
- String volumeName, String bucketName, String keyName) {
+ String volumeName, String bucketName, String keyName) throws Exception {
OMRequest omRequest =
TestOMRequestUtils.createInitiateMPURequest(volumeName, bucketName,
keyName);
@@ -141,7 +141,8 @@ public class TestS3MultipartRequest {
*/
protected OMRequest doPreExecuteCommitMPU(
String volumeName, String bucketName, String keyName,
- long clientID, String multipartUploadID, int partNumber) {
+ long clientID, String multipartUploadID, int partNumber)
+ throws Exception {
// Just set dummy size
long dataSize = 100L;
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java
index 5b220bf..d623b17 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCommitPartRequest.java
@@ -38,7 +38,7 @@ public class TestS3MultipartUploadCommitPartRequest
extends TestS3MultipartRequest {
@Test
- public void testPreExecute() {
+ public void testPreExecute() throws Exception {
String volumeName = UUID.randomUUID().toString();
String bucketName = UUID.randomUUID().toString();
String keyName = UUID.randomUUID().toString();
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 11/39: HDDS-3798. Display version and setupTime of
DN in recon web (#1136)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit f566187d9c9714b668be6fb06c5a950cad37d1a9
Author: HuangTao <th...@163.com>
AuthorDate: Wed Jul 15 02:50:20 2020 +0800
HDDS-3798. Display version and setupTime of DN in recon web (#1136)
(cherry picked from commit 4d964f3f3d4e97e5e0131cd6edd048f5fffc72fa)
---
.gitignore | 2 +
.../hadoop/hdds/protocol/DatanodeDetails.java | 89 +++++++++++++++++-
.../apache/hadoop/ozone/HddsDatanodeService.java | 5 ++
.../interface-client/src/main/proto/hdds.proto | 2 +
.../interface-client/src/main/proto/proto.lock | 12 ++-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 2 +
.../ozone/recon/api/types/DatanodeMetadata.java | 28 ++++++
.../webapps/recon/ozone-recon-web/api/db.json | 48 +++++++---
.../src/components/multiSelect/multiSelect.tsx | 5 +-
.../src/views/datanodes/datanodes.less | 14 +++
.../src/views/datanodes/datanodes.tsx | 100 +++++++++++++++++++--
11 files changed, 285 insertions(+), 22 deletions(-)
diff --git a/.gitignore b/.gitignore
index 551b1b5..e09c2eb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -14,6 +14,7 @@
.classpath
.project
.settings
+*.factorypath
target
build
dependency-reduced-pom.xml
@@ -61,5 +62,6 @@ output.xml
report.html
hadoop-hdds/docs/public
+hadoop-ozone/recon/node_modules
.mvn
diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
index 1b6a214..96f19a6 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
@@ -49,6 +49,8 @@ public class DatanodeDetails extends NodeImpl implements
private String hostName;
private List<Port> ports;
private String certSerialId;
+ private String version;
+ private long setupTime;
/**
* Constructs DatanodeDetails instance. DatanodeDetails.Builder is used
@@ -59,15 +61,21 @@ public class DatanodeDetails extends NodeImpl implements
* @param networkLocation DataNode's network location path
* @param ports Ports used by the DataNode
* @param certSerialId serial id from SCM issued certificate.
+ * @param version DataNode's version
+ * @param setupTime the setup time of DataNode
*/
+ @SuppressWarnings("parameternumber")
private DatanodeDetails(UUID uuid, String ipAddress, String hostName,
- String networkLocation, List<Port> ports, String certSerialId) {
+ String networkLocation, List<Port> ports, String certSerialId,
+ String version, long setupTime) {
super(hostName, networkLocation, NetConstants.NODE_COST_DEFAULT);
this.uuid = uuid;
this.ipAddress = ipAddress;
this.hostName = hostName;
this.ports = ports;
this.certSerialId = certSerialId;
+ this.version = version;
+ this.setupTime = setupTime;
}
public DatanodeDetails(DatanodeDetails datanodeDetails) {
@@ -79,6 +87,8 @@ public class DatanodeDetails extends NodeImpl implements
this.ports = datanodeDetails.ports;
this.setNetworkName(datanodeDetails.getNetworkName());
this.setParent(datanodeDetails.getParent());
+ this.version = datanodeDetails.version;
+ this.setupTime = datanodeDetails.setupTime;
}
/**
@@ -207,6 +217,12 @@ public class DatanodeDetails extends NodeImpl implements
if (datanodeDetailsProto.hasNetworkLocation()) {
builder.setNetworkLocation(datanodeDetailsProto.getNetworkLocation());
}
+ if (datanodeDetailsProto.hasVersion()) {
+ builder.setVersion(datanodeDetailsProto.getVersion());
+ }
+ if (datanodeDetailsProto.hasSetupTime()) {
+ builder.setSetupTime(datanodeDetailsProto.getSetupTime());
+ }
return builder.build();
}
@@ -248,6 +264,13 @@ public class DatanodeDetails extends NodeImpl implements
.setValue(port.getValue())
.build());
}
+
+ if (!Strings.isNullOrEmpty(getVersion())) {
+ builder.setVersion(getVersion());
+ }
+
+ builder.setSetupTime(getSetupTime());
+
return builder.build();
}
@@ -300,6 +323,8 @@ public class DatanodeDetails extends NodeImpl implements
private String networkLocation;
private List<Port> ports;
private String certSerialId;
+ private String version;
+ private long setupTime;
/**
* Default private constructor. To create Builder instance use
@@ -389,6 +414,30 @@ public class DatanodeDetails extends NodeImpl implements
}
/**
+ * Sets the DataNode version.
+ *
+ * @param ver the version of DataNode.
+ *
+ * @return DatanodeDetails.Builder
+ */
+ public Builder setVersion(String ver) {
+ this.version = ver;
+ return this;
+ }
+
+ /**
+ * Sets the DataNode setup time.
+ *
+ * @param time the setup time of DataNode.
+ *
+ * @return DatanodeDetails.Builder
+ */
+ public Builder setSetupTime(long time) {
+ this.setupTime = time;
+ return this;
+ }
+
+ /**
* Builds and returns DatanodeDetails instance.
*
* @return DatanodeDetails
@@ -399,7 +448,7 @@ public class DatanodeDetails extends NodeImpl implements
networkLocation = NetConstants.DEFAULT_RACK;
}
DatanodeDetails dn = new DatanodeDetails(id, ipAddress, hostName,
- networkLocation, ports, certSerialId);
+ networkLocation, ports, certSerialId, version, setupTime);
if (networkName != null) {
dn.setNetworkName(networkName);
}
@@ -505,4 +554,40 @@ public class DatanodeDetails extends NodeImpl implements
public void setCertSerialId(String certSerialId) {
this.certSerialId = certSerialId;
}
+
+ /**
+ * Returns the DataNode version.
+ *
+ * @return DataNode version
+ */
+ public String getVersion() {
+ return version;
+ }
+
+ /**
+ * Set DataNode version.
+ *
+ * @param version DataNode version
+ */
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ /**
+ * Returns the DataNode setup time.
+ *
+ * @return DataNode setup time
+ */
+ public long getSetupTime() {
+ return setupTime;
+ }
+
+ /**
+ * Set DataNode setup time.
+ *
+ * @param setupTime DataNode setup time
+ */
+ public void setSetupTime(long setupTime) {
+ this.setupTime = setupTime;
+ }
}
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
index 7e896e7..08eef6f 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
@@ -65,6 +65,8 @@ import static org.apache.hadoop.hdds.security.x509.certificate.utils.Certificate
import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
import static org.apache.hadoop.ozone.OzoneConfigKeys.HDDS_DATANODE_PLUGINS_KEY;
import static org.apache.hadoop.util.ExitUtil.terminate;
+
+import org.apache.hadoop.util.Time;
import org.bouncycastle.pkcs.PKCS10CertificationRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -191,6 +193,9 @@ public class HddsDatanodeService extends GenericCli implements ServicePlugin {
datanodeDetails = initializeDatanodeDetails();
datanodeDetails.setHostName(hostname);
datanodeDetails.setIpAddress(ip);
+ datanodeDetails.setVersion(
+ HddsVersionInfo.HDDS_VERSION_INFO.getVersion());
+ datanodeDetails.setSetupTime(Time.now());
TracingUtil.initTracing(
"HddsDatanodeService." + datanodeDetails.getUuidString()
.substring(0, 8), conf);
diff --git a/hadoop-hdds/interface-client/src/main/proto/hdds.proto b/hadoop-hdds/interface-client/src/main/proto/hdds.proto
index 23cc9cb..243e8ec 100644
--- a/hadoop-hdds/interface-client/src/main/proto/hdds.proto
+++ b/hadoop-hdds/interface-client/src/main/proto/hdds.proto
@@ -43,6 +43,8 @@ message DatanodeDetailsProto {
// network name, can be Ip address or host name, depends
optional string networkName = 6;
optional string networkLocation = 7; // Network topology location
+ optional string version = 8; // Datanode version
+ optional int64 setupTime = 9;
// TODO(runzhiwang): when uuid is gone, specify 1 as the index of uuid128 and mark as required
optional UUID uuid128 = 100; // UUID with 128 bits assigned to the Datanode.
}
diff --git a/hadoop-hdds/interface-client/src/main/proto/proto.lock b/hadoop-hdds/interface-client/src/main/proto/proto.lock
index afdaf96..b27896c 100644
--- a/hadoop-hdds/interface-client/src/main/proto/proto.lock
+++ b/hadoop-hdds/interface-client/src/main/proto/proto.lock
@@ -1531,6 +1531,16 @@
"type": "string"
},
{
+ "id": 8,
+ "name": "version",
+ "type": "string"
+ },
+ {
+ "id": 9,
+ "name": "setupTime",
+ "type": "int64"
+ },
+ {
"id": 100,
"name": "uuid128",
"type": "UUID"
@@ -1925,4 +1935,4 @@
}
}
]
-}
\ No newline at end of file
+}
diff --git a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
index 2c01749..42832de 100644
--- a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
+++ b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
@@ -121,6 +121,8 @@ public class NodeEndpoint {
.withPipelines(pipelines)
.withLeaderCount(leaderCount.get())
.withUUid(datanode.getUuidString())
+ .withVersion(datanode.getVersion())
+ .withSetupTime(datanode.getSetupTime())
.build());
});
diff --git a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeMetadata.java b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeMetadata.java
index 02d9ae8..542654e 100644
--- a/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeMetadata.java
+++ b/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodeMetadata.java
@@ -55,6 +55,12 @@ public final class DatanodeMetadata {
@XmlElement(name = "leaderCount")
private int leaderCount;
+ @XmlElement(name = "version")
+ private String version;
+
+ @XmlElement(name = "setupTime")
+ private long setupTime;
+
private DatanodeMetadata(Builder builder) {
this.hostname = builder.hostname;
this.uuid = builder.uuid;
@@ -64,6 +70,8 @@ public final class DatanodeMetadata {
this.pipelines = builder.pipelines;
this.containers = builder.containers;
this.leaderCount = builder.leaderCount;
+ this.version = builder.version;
+ this.setupTime = builder.setupTime;
}
public String getHostname() {
@@ -98,6 +106,14 @@ public final class DatanodeMetadata {
return uuid;
}
+ public String getVersion() {
+ return version;
+ }
+
+ public long getSetupTime() {
+ return setupTime;
+ }
+
/**
* Returns new builder class that builds a DatanodeMetadata.
*
@@ -120,6 +136,8 @@ public final class DatanodeMetadata {
private List<DatanodePipeline> pipelines;
private int containers;
private int leaderCount;
+ private String version;
+ private long setupTime;
public Builder() {
this.containers = 0;
@@ -167,6 +185,16 @@ public final class DatanodeMetadata {
return this;
}
+ public Builder withVersion(String version) {
+ this.version = version;
+ return this;
+ }
+
+ public Builder withSetupTime(long setupTime) {
+ this.setupTime = setupTime;
+ return this;
+ }
+
/**
* Constructs DatanodeMetadata.
*
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/api/db.json b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/api/db.json
index 82fae37..d8d6eac 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/api/db.json
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/api/db.json
@@ -41,7 +41,9 @@
}
],
"containers": 80,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574728775759
},
{
"hostname": "localhost2.storage.enterprise.com",
@@ -68,7 +70,9 @@
}
],
"containers": 8192,
- "leaderCount": 1
+ "leaderCount": 1,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724805059
},
{
"hostname": "localhost3.storage.enterprise.com",
@@ -101,7 +105,9 @@
}
],
"containers": 43,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1343544679543
},
{
"hostname": "localhost4.storage.enterprise.com",
@@ -115,7 +121,9 @@
},
"pipelines": [],
"containers": 0,
- "leaderCount": 0
+ "leaderCount": 0,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1074724802059
},
{
"hostname": "localhost5.storage.enterprise.com",
@@ -142,7 +150,9 @@
}
],
"containers": 643,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724816029
},
{
"hostname": "localhost6.storage.enterprise.com",
@@ -169,7 +179,9 @@
}
],
"containers": 5,
- "leaderCount": 1
+ "leaderCount": 1,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724802059
},
{
"hostname": "localhost7.storage.enterprise.com",
@@ -202,7 +214,9 @@
}
],
"containers": 64,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724676009
},
{
"hostname": "localhost8.storage.enterprise.com",
@@ -229,7 +243,9 @@
}
],
"containers": 21,
- "leaderCount": 1
+ "leaderCount": 1,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724276050
},
{
"hostname": "localhost9.storage.enterprise.com",
@@ -256,7 +272,9 @@
}
],
"containers": 897,
- "leaderCount": 1
+ "leaderCount": 1,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724573011
},
{
"hostname": "localhost10.storage.enterprise.com",
@@ -289,7 +307,9 @@
}
],
"containers": 6754,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574723756059
},
{
"hostname": "localhost11.storage.enterprise.com",
@@ -316,7 +336,9 @@
}
],
"containers": 78,
- "leaderCount": 2
+ "leaderCount": 2,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1474724705783
},
{
"hostname": "localhost12.storage.enterprise.com",
@@ -343,7 +365,9 @@
}
],
"containers": 543,
- "leaderCount": 1
+ "leaderCount": 1,
+ "version": "0.6.0-SNAPSHOT",
+ "setupTime": 1574724706232
}
]
},
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/multiSelect/multiSelect.tsx b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/multiSelect/multiSelect.tsx
index 19005dd..417c2ef 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/multiSelect/multiSelect.tsx
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/multiSelect/multiSelect.tsx
@@ -36,6 +36,7 @@ interface IMultiSelectProps extends ReactSelectProps<IOption> {
options: IOption[];
allowSelectAll: boolean;
allOption?: IOption;
+ maxShowValues?: number;
}
const defaultProps = {
@@ -48,7 +49,7 @@ const defaultProps = {
export class MultiSelect extends PureComponent<IMultiSelectProps> {
static defaultProps = defaultProps;
render() {
- const {allowSelectAll, allOption, options, onChange} = this.props;
+ const {allowSelectAll, allOption, options, maxShowValues = 5, onChange} = this.props;
if (allowSelectAll) {
const Option = (props: OptionProps<IOption>) => {
return (
@@ -70,7 +71,7 @@ export class MultiSelect extends PureComponent<IMultiSelectProps> {
let toBeRendered = children;
if (currentValues.some(val => val.value === allOption!.value) && children) {
toBeRendered = allOption!.label;
- } else if (currentValues.length >= 5) {
+ } else if (currentValues.length > maxShowValues) {
toBeRendered = `${currentValues.length} selected`;
}
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.less b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.less
index 4a3cdf5..10ec907 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.less
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.less
@@ -22,4 +22,18 @@
margin-bottom: 5px;
}
}
+
+ .filter-block {
+ font-size: 14px;
+ font-weight: normal;
+ display: inline-block;
+ margin-left: 20px;
+ }
+
+ .multi-select-container {
+ padding-left: 5px;
+ margin-right: 5px;
+ display: inline-block;
+ min-width: 200px;
+ }
}
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
index bfba82a..877ebf9 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
@@ -27,6 +27,8 @@ import {DatanodeStatus, IStorageReport} from 'types/datanode.types';
import './datanodes.less';
import {AutoReloadHelper} from 'utils/autoReloadHelper';
import AutoReloadPanel from 'components/autoReloadPanel/autoReloadPanel';
+import {MultiSelect, IOption} from 'components/multiSelect/multiSelect';
+import {ActionMeta, ValueType} from 'react-select';
import {showDataFetchError} from 'utils/common';
interface IDatanodeResponse {
@@ -38,6 +40,8 @@ interface IDatanodeResponse {
containers: number;
leaderCount: number;
uuid: string;
+ version: string;
+ setupTime: number;
}
interface IDatanodesResponse {
@@ -56,6 +60,8 @@ interface IDatanode {
containers: number;
leaderCount: number;
uuid: string;
+ version: string;
+ setupTime: number;
}
interface IPipeline {
@@ -70,6 +76,8 @@ interface IDatanodesState {
dataSource: IDatanode[];
totalCount: number;
lastUpdated: number;
+ selectedColumns: IOption[];
+ columnOptions: IOption[];
}
const renderDatanodeStatus = (status: DatanodeStatus) => {
@@ -89,6 +97,7 @@ const COLUMNS = [
title: 'Status',
dataIndex: 'state',
key: 'state',
+ isVisible: true,
render: (text: DatanodeStatus) => renderDatanodeStatus(text),
sorter: (a: IDatanode, b: IDatanode) => a.state.localeCompare(b.state)
},
@@ -96,6 +105,7 @@ const COLUMNS = [
title: 'Uuid',
dataIndex: 'uuid',
key: 'uuid',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.uuid.localeCompare(b.uuid),
defaultSortOrder: 'ascend' as const
},
@@ -103,6 +113,7 @@ const COLUMNS = [
title: 'Hostname',
dataIndex: 'hostname',
key: 'hostname',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.hostname.localeCompare(b.hostname),
defaultSortOrder: 'ascend' as const
},
@@ -110,6 +121,7 @@ const COLUMNS = [
title: 'Storage Capacity',
dataIndex: 'storageUsed',
key: 'storageUsed',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.storageRemaining - b.storageRemaining,
render: (text: string, record: IDatanode) => (
<StorageBar
@@ -120,6 +132,7 @@ const COLUMNS = [
title: 'Last Heartbeat',
dataIndex: 'lastHeartbeat',
key: 'lastHeartbeat',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.lastHeartbeat - b.lastHeartbeat,
render: (heartbeat: number) => {
return heartbeat > 0 ? moment(heartbeat).format('lll') : 'NA';
@@ -129,6 +142,7 @@ const COLUMNS = [
title: 'Pipeline ID(s)',
dataIndex: 'pipelines',
key: 'pipelines',
+ isVisible: true,
render: (pipelines: IPipeline[], record: IDatanode) => {
return (
<div>
@@ -158,16 +172,46 @@ const COLUMNS = [
</span>,
dataIndex: 'leaderCount',
key: 'leaderCount',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.leaderCount - b.leaderCount
},
{
title: 'Containers',
dataIndex: 'containers',
key: 'containers',
+ isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.containers - b.containers
+ },
+ {
+ title: 'Version',
+ dataIndex: 'version',
+ key: 'version',
+ isVisible: false,
+ sorter: (a: IDatanode, b: IDatanode) => a.version.localeCompare(b.version),
+ defaultSortOrder: 'ascend' as const
+ },
+ {
+ title: 'SetupTime',
+ dataIndex: 'setupTime',
+ key: 'setupTime',
+ isVisible: false,
+ sorter: (a: IDatanode, b: IDatanode) => a.setupTime - b.setupTime,
+ render: (uptime: number) => {
+ return uptime > 0 ? moment(uptime).format('lll') : 'NA';
+ }
}
];
+const allColumnsOption: IOption = {
+ label: 'Select all',
+ value: '*'
+};
+
+const defaultColumns: IOption[] = COLUMNS.map(column => ({
+ label: column.key,
+ value: column.key
+}));
+
export class Datanodes extends React.Component<Record<string, object>, IDatanodesState> {
autoReload: AutoReloadHelper;
@@ -177,11 +221,20 @@ export class Datanodes extends React.Component<Record<string, object>, IDatanode
loading: false,
dataSource: [],
totalCount: 0,
- lastUpdated: 0
+ lastUpdated: 0,
+ selectedColumns: [],
+ columnOptions: defaultColumns
};
this.autoReload = new AutoReloadHelper(this._loadData);
}
+ _handleColumnChange = (selected: ValueType<IOption>, _action: ActionMeta<IOption>) => {
+ const selectedColumns = (selected as IOption[]);
+ this.setState({
+ selectedColumns
+ });
+ };
+
_loadData = () => {
this.setState({
loading: true
@@ -201,14 +254,23 @@ export class Datanodes extends React.Component<Record<string, object>, IDatanode
storageRemaining: datanode.storageReport.remaining,
pipelines: datanode.pipelines,
containers: datanode.containers,
- leaderCount: datanode.leaderCount
+ leaderCount: datanode.leaderCount,
+ version: datanode.version,
+ setupTime: datanode.setupTime
};
});
+ const selectedColumns: IOption[] = COLUMNS.filter(column => column.isVisible).map(column => ({
+ label: column.key,
+ value: column.key
+ }));
+
this.setState({
loading: false,
dataSource,
totalCount,
lastUpdated: Number(moment())
+ }, () => {
+ this._handleColumnChange(selectedColumns, {action: 'select-option'});
});
}).catch(error => {
this.setState({
@@ -233,7 +295,7 @@ export class Datanodes extends React.Component<Record<string, object>, IDatanode
};
render() {
- const {dataSource, loading, totalCount, lastUpdated} = this.state;
+ const {dataSource, loading, totalCount, lastUpdated, selectedColumns, columnOptions} = this.state;
const paginationConfig: PaginationConfig = {
showTotal: (total: number, range) => `${range[0]}-${range[1]} of ${total} datanodes`,
showSizeChanger: true,
@@ -243,10 +305,38 @@ export class Datanodes extends React.Component<Record<string, object>, IDatanode
<div className='datanodes-container'>
<div className='page-header'>
Datanodes ({totalCount})
- <AutoReloadPanel isLoading={loading} lastUpdated={lastUpdated} togglePolling={this.autoReload.handleAutoReloadToggle} onReload={this._loadData}/>
+ <div className='filter-block'>
+ <MultiSelect
+ allowSelectAll
+ isMulti
+ maxShowValues={3}
+ className='multi-select-container'
+ options={columnOptions}
+ closeMenuOnSelect={false}
+ hideSelectedOptions={false}
+ value={selectedColumns}
+ allOption={allColumnsOption}
+ onChange={this._handleColumnChange}
+ /> Columns
+ </div>
+ <AutoReloadPanel
+ isLoading={loading}
+ lastUpdated={lastUpdated}
+ togglePolling={this.autoReload.handleAutoReloadToggle}
+ onReload={this._loadData}
+ />
</div>
+
<div className='content-div'>
- <Table dataSource={dataSource} columns={COLUMNS} loading={loading} pagination={paginationConfig} rowKey='hostname'/>
+ <Table
+ dataSource={dataSource}
+ columns={COLUMNS.filter(column =>
+ selectedColumns.some(e => e.value === column.key)
+ )}
+ loading={loading}
+ pagination={paginationConfig}
+ rowKey='hostname'
+ />
</div>
</div>
);
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 33/39: HDDS-3982. Disable moveToTrash in o3fs and
ofs temporarily (#1215)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit a90b7bba1599c9cd85081fb573dcb4fc64d1fada
Author: Siyao Meng <50...@users.noreply.github.com>
AuthorDate: Mon Jul 20 20:11:23 2020 -0700
HDDS-3982. Disable moveToTrash in o3fs and ofs temporarily (#1215)
(cherry picked from commit fbd125cd7cb1aea7941d251d31772ba3f37696b4)
---
hadoop-hdds/docs/content/design/ofs.md | 4 +++
hadoop-hdds/docs/content/design/trash.md | 7 +++-
hadoop-hdds/docs/content/interface/OzoneFS.md | 3 ++
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 38 ++++++++++++++++++++++
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 37 +++++++++++++++++++++
.../hadoop/ozone/shell/TestOzoneShellHA.java | 2 ++
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 29 +++++++++++++++++
.../fs/ozone/BasicRootedOzoneFileSystem.java | 29 +++++++++++++++++
8 files changed, 148 insertions(+), 1 deletion(-)
diff --git a/hadoop-hdds/docs/content/design/ofs.md b/hadoop-hdds/docs/content/design/ofs.md
index 9a2352e..7a39665 100644
--- a/hadoop-hdds/docs/content/design/ofs.md
+++ b/hadoop-hdds/docs/content/design/ofs.md
@@ -155,6 +155,10 @@ This feature wouldn't degrade server performance as the loop is on the client.
Think it as a client is issuing multiple requests to the server to get all the
information.
+# Special note
+
+Trash is disabled even if `fs.trash.interval` is set on purpose. (HDDS-3982)
+
# Link
Design doc is uploaded to the JIRA HDDS-2665:
diff --git a/hadoop-hdds/docs/content/design/trash.md b/hadoop-hdds/docs/content/design/trash.md
index 78e077a..b936aae 100644
--- a/hadoop-hdds/docs/content/design/trash.md
+++ b/hadoop-hdds/docs/content/design/trash.md
@@ -22,4 +22,9 @@ author: Matthew Sharp
The design doc is uploaded to the JIRA:
-https://issues.apache.org/jira/secure/attachment/12985273/Ozone_Trash_Feature.docx
\ No newline at end of file
+https://issues.apache.org/jira/secure/attachment/12985273/Ozone_Trash_Feature.docx
+
+## Special note
+
+Trash is disabled for both o3fs and ofs even if `fs.trash.interval` is set
+on purpose. (HDDS-3982)
diff --git a/hadoop-hdds/docs/content/interface/OzoneFS.md b/hadoop-hdds/docs/content/interface/OzoneFS.md
index 98bf2f9..7aebe4f 100644
--- a/hadoop-hdds/docs/content/interface/OzoneFS.md
+++ b/hadoop-hdds/docs/content/interface/OzoneFS.md
@@ -165,3 +165,6 @@ hdfs dfs -put /etc/hosts /volume1/bucket1/test
For more usage, see: https://issues.apache.org/jira/secure/attachment/12987636/Design%20ofs%20v1.pdf
+## Special note
+
+Trash is disabled even if `fs.trash.interval` is set on purpose. (HDDS-3982)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystem.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystem.java
index 700506a..ad1705a 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystem.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystem.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.Trash;
import org.apache.hadoop.fs.contract.ContractTestUtils;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.MiniOzoneCluster;
@@ -47,6 +48,7 @@ import org.apache.hadoop.test.GenericTestUtils;
import org.apache.commons.io.IOUtils;
+import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY;
import static org.apache.hadoop.fs.FileSystem.TRASH_PREFIX;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_FS_ITERATE_BATCH_SIZE;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER;
@@ -86,6 +88,7 @@ public class TestOzoneFileSystem {
private String volumeName;
private String bucketName;
private int rootItemCount;
+ private Trash trash;
@Test(timeout = 300_000)
public void testCreateFileShouldCheckExistenceOfDirWithSameName()
@@ -167,6 +170,8 @@ public class TestOzoneFileSystem {
testOzoneFsServiceLoader();
o3fs = (OzoneFileSystem) fs;
+ testRenameToTrashDisabled();
+
testGetTrashRoots();
testGetTrashRoot();
testGetDirectoryModificationTime();
@@ -197,6 +202,7 @@ public class TestOzoneFileSystem {
private void setupOzoneFileSystem()
throws IOException, TimeoutException, InterruptedException {
OzoneConfiguration conf = new OzoneConfiguration();
+ conf.setInt(FS_TRASH_INTERVAL_KEY, 1);
cluster = MiniOzoneCluster.newBuilder(conf)
.setNumDatanodes(3)
.build();
@@ -215,6 +221,7 @@ public class TestOzoneFileSystem {
// Set the number of keys to be processed during batch operate.
conf.setInt(OZONE_FS_ITERATE_BATCH_SIZE, 5);
fs = FileSystem.get(conf);
+ trash = new Trash(conf);
}
private void testOzoneFsServiceLoader() throws IOException {
@@ -617,4 +624,35 @@ public class TestOzoneFileSystem {
// Clean up
o3fs.delete(trashRoot, true);
}
+
+ /**
+ * Check that no files are actually moved to trash since it is disabled by
+ * fs.rename(src, dst, options).
+ */
+ public void testRenameToTrashDisabled() throws IOException {
+ // Create a file
+ String testKeyName = "testKey1";
+ Path path = new Path(OZONE_URI_DELIMITER, testKeyName);
+ try (FSDataOutputStream stream = fs.create(path)) {
+ stream.write(1);
+ }
+
+ // Call moveToTrash. We can't call protected fs.rename() directly
+ trash.moveToTrash(path);
+
+ // Construct paths
+ String username = UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(OZONE_URI_DELIMITER, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, "Current");
+ Path trashPath = new Path(userTrashCurrent, testKeyName);
+
+ // Trash Current directory should still have been created.
+ Assert.assertTrue(o3fs.exists(userTrashCurrent));
+ // Check under trash, the key should be deleted instead
+ Assert.assertFalse(o3fs.exists(trashPath));
+
+ // Cleanup
+ o3fs.delete(trashRoot, true);
+ }
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
index 8c71c61..3aec3e8 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
@@ -26,6 +26,7 @@ import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
+import org.apache.hadoop.fs.Trash;
import org.apache.hadoop.fs.contract.ContractTestUtils;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.hdds.conf.OzoneConfiguration;
@@ -64,6 +65,7 @@ import java.util.Set;
import java.util.TreeSet;
import java.util.stream.Collectors;
+import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_TRASH_INTERVAL_KEY;
import static org.apache.hadoop.fs.FileSystem.TRASH_PREFIX;
import static org.apache.hadoop.fs.ozone.Constants.LISTING_PAGE_SIZE;
import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
@@ -87,6 +89,7 @@ public class TestRootedOzoneFileSystem {
private RootedOzoneFileSystem ofs;
private ObjectStore objectStore;
private static BasicRootedOzoneClientAdapterImpl adapter;
+ private Trash trash;
private String volumeName;
private Path volumePath;
@@ -98,6 +101,7 @@ public class TestRootedOzoneFileSystem {
@Before
public void init() throws Exception {
conf = new OzoneConfiguration();
+ conf.setInt(FS_TRASH_INTERVAL_KEY, 1);
cluster = MiniOzoneCluster.newBuilder(conf)
.setNumDatanodes(3)
.build();
@@ -122,6 +126,7 @@ public class TestRootedOzoneFileSystem {
// hence this workaround.
conf.set("fs.ofs.impl", "org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
fs = FileSystem.get(conf);
+ trash = new Trash(conf);
ofs = (RootedOzoneFileSystem) fs;
adapter = (BasicRootedOzoneClientAdapterImpl) ofs.getAdapter();
}
@@ -999,4 +1004,36 @@ public class TestRootedOzoneFileSystem {
Assert.assertTrue(volume1.setOwner(prevOwner));
}
+ /**
+ * Check that no files are actually moved to trash since it is disabled by
+ * fs.rename(src, dst, options).
+ */
+ @Test
+ public void testRenameToTrashDisabled() throws IOException {
+ // Create a file
+ String testKeyName = "testKey1";
+ Path path = new Path(bucketPath, testKeyName);
+ try (FSDataOutputStream stream = fs.create(path)) {
+ stream.write(1);
+ }
+
+ // Call moveToTrash. We can't call protected fs.rename() directly
+ trash.moveToTrash(path);
+
+ // Construct paths
+ String username = UserGroupInformation.getCurrentUser().getShortUserName();
+ Path trashRoot = new Path(bucketPath, TRASH_PREFIX);
+ Path userTrash = new Path(trashRoot, username);
+ Path userTrashCurrent = new Path(userTrash, "Current");
+ Path trashPath = new Path(userTrashCurrent, testKeyName);
+
+ // Trash Current directory should still have been created.
+ Assert.assertTrue(ofs.exists(userTrashCurrent));
+ // Check under trash, the key should be deleted instead
+ Assert.assertFalse(ofs.exists(trashPath));
+
+ // Cleanup
+ ofs.delete(trashRoot, true);
+ }
+
}
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
index e340d40..17baa06 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
@@ -38,6 +38,7 @@ import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;
@@ -468,6 +469,7 @@ public class TestOzoneShellHA {
}
@Test
+ @Ignore("HDDS-3982. Disable moveToTrash in o3fs and ofs temporarily")
public void testDeleteToTrashOrSkipTrash() throws Exception {
final String hostPrefix = OZONE_OFS_URI_SCHEME + "://" + omServiceId;
OzoneConfiguration clientConf = getClientConfForOFS(hostPrefix, conf);
diff --git a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
index 25dea3d..778277f 100644
--- a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
+++ b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
@@ -29,6 +29,7 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Options.Rename;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
import org.apache.hadoop.fs.permission.FsPermission;
@@ -397,6 +398,34 @@ public class BasicOzoneFileSystem extends FileSystem {
return result;
}
+ /**
+ * Intercept rename to trash calls from TrashPolicyDefault,
+ * convert them to delete calls instead.
+ */
+ @Deprecated
+ protected void rename(final Path src, final Path dst,
+ final Rename... options) throws IOException {
+ boolean hasMoveToTrash = false;
+ if (options != null) {
+ for (Rename option : options) {
+ if (option == Rename.TO_TRASH) {
+ hasMoveToTrash = true;
+ break;
+ }
+ }
+ }
+ if (!hasMoveToTrash) {
+ // if doesn't have TO_TRASH option, just pass the call to super
+ super.rename(src, dst, options);
+ } else {
+ // intercept when TO_TRASH is found
+ LOG.info("Move to trash is disabled for o3fs, deleting instead: {}. "
+ + "Files or directories will NOT be retained in trash. "
+ + "Ignore the following TrashPolicyDefault message, if any.", src);
+ delete(src, true);
+ }
+ }
+
private class DeleteIterator extends OzoneListingIterator {
private boolean recursive;
diff --git a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
index 7c28428..3bea2bb 100644
--- a/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
+++ b/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java
@@ -29,6 +29,7 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Options;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
import org.apache.hadoop.fs.permission.FsPermission;
@@ -372,6 +373,34 @@ public class BasicRootedOzoneFileSystem extends FileSystem {
return result;
}
+ /**
+ * Intercept rename to trash calls from TrashPolicyDefault,
+ * convert them to delete calls instead.
+ */
+ @Deprecated
+ protected void rename(final Path src, final Path dst,
+ final Options.Rename... options) throws IOException {
+ boolean hasMoveToTrash = false;
+ if (options != null) {
+ for (Options.Rename option : options) {
+ if (option == Options.Rename.TO_TRASH) {
+ hasMoveToTrash = true;
+ break;
+ }
+ }
+ }
+ if (!hasMoveToTrash) {
+ // if doesn't have TO_TRASH option, just pass the call to super
+ super.rename(src, dst, options);
+ } else {
+ // intercept when TO_TRASH is found
+ LOG.info("Move to trash is disabled for ofs, deleting instead: {}. "
+ + "Files or directories will NOT be retained in trash. "
+ + "Ignore the following TrashPolicyDefault message, if any.", src);
+ delete(src, true);
+ }
+ }
+
private class DeleteIterator extends OzoneListingIterator {
final private boolean recursive;
private final OzoneBucket bucket;
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 16/39: HDDS-2769. security/SecurityWithRanger.md
(#1189)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 06e186bfe1b7898ec5ba72a3a4045357fabcfe79
Author: Xiang Zhang <xi...@gmail.com>
AuthorDate: Wed Jul 15 14:54:02 2020 +0800
HDDS-2769. security/SecurityWithRanger.md (#1189)
(cherry picked from commit 926048403d115ddcb59ff130e5c46e518874b8aa)
---
.../docs/content/security/SecuityWithRanger.zh.md | 35 ++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/hadoop-hdds/docs/content/security/SecuityWithRanger.zh.md b/hadoop-hdds/docs/content/security/SecuityWithRanger.zh.md
new file mode 100644
index 0000000..b879e93
--- /dev/null
+++ b/hadoop-hdds/docs/content/security/SecuityWithRanger.zh.md
@@ -0,0 +1,35 @@
+---
+title: "Apache Ranger"
+date: "2019-April-03"
+weight: 5
+summary: Apache Ranger 是一个用于管理和监控 Hadoop 平台复杂数据权限的框架。
+icon: user
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+Apache Ranger™ 是一个用于管理和监控 Hadoop 平台复杂数据权限的框架。版本大于 1.20 的 Apache Ranger 都可以用于管理 Ozone 集群。
+
+你需要先在你的 Hadoop 集群上安装 Apache Ranger，安装指南可以参考 [Apache Ranger 官网](https://ranger.apache.org/index.html).
+
+如果你已经安装好了 Apache Ranger，那么 Ozone 的配置十分简单，你只需要启用 ACL 支持并且将 ACL 授权类设置为 Ranger 授权类，在 ozone-site.xml 中添加下面的参数：
+
+参数名|参数值
+--------|------------------------------------------------------------
+ozone.acl.enabled | true
+ozone.acl.authorizer.class| org.apache.ranger.authorization.ozone.authorizer.RangerOzoneAuthorizer
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 04/39: HDDS-3941. Enable core dump when crash in C++
(#1186)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 9c9daa16486b1806c9ad122721d806da61648b9a
Author: runzhiwang <51...@users.noreply.github.com>
AuthorDate: Mon Jul 13 14:41:46 2020 +0800
HDDS-3941. Enable core dump when crash in C++ (#1186)
(cherry picked from commit c9f6365e3e17dbdaaa367478bcc5d8390f3ee41f)
---
hadoop-hdds/common/src/main/conf/hadoop-env.sh | 3 +++
1 file changed, 3 insertions(+)
diff --git a/hadoop-hdds/common/src/main/conf/hadoop-env.sh b/hadoop-hdds/common/src/main/conf/hadoop-env.sh
index e43cd95..51ee585 100644
--- a/hadoop-hdds/common/src/main/conf/hadoop-env.sh
+++ b/hadoop-hdds/common/src/main/conf/hadoop-env.sh
@@ -30,6 +30,9 @@
## {YARN_xyz|HDFS_xyz} > HADOOP_xyz > hard-coded defaults
##
+# Enable core dump when crash in C++
+ulimit -c unlimited
+
# Many of the options here are built from the perspective that users
# may want to provide OVERWRITING values on the command line.
# For example:
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 23/39: HDDS-3807. Propagate raft log disks info to
SCM from datanode. (#1107)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 82657a7ea1c66c39d9a7bf409be2ea96728e0d03
Author: bshashikant <sh...@apache.org>
AuthorDate: Thu Jul 16 16:08:56 2020 +0530
HDDS-3807. Propagate raft log disks info to SCM from datanode. (#1107)
(cherry picked from commit de027855798bf3b891b8d3c00dc8e59531f98781)
---
.../common/impl/StorageLocationReport.java | 11 ++--
.../common/transport/server/XceiverServerSpi.java | 12 ++++-
.../transport/server/ratis/XceiverServerRatis.java | 58 ++++++++++++++++++++--
.../container/common/volume/MutableVolumeSet.java | 10 +---
.../ozone/container/ozoneimpl/OzoneContainer.java | 18 ++++++-
.../container/ozoneimpl/TestOzoneContainer.java | 51 +++++++++++++++++--
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 16 +++---
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 6 +++
8 files changed, 152 insertions(+), 30 deletions(-)
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/StorageLocationReport.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/StorageLocationReport.java
index 061d09b..2ad7f0d 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/StorageLocationReport.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/StorageLocationReport.java
@@ -88,11 +88,14 @@ public final class StorageLocationReport implements
return storageType;
}
+ private StorageTypeProto getStorageTypeProto() throws IOException {
+ return getStorageTypeProto(getStorageType());
+ }
- private StorageTypeProto getStorageTypeProto() throws
- IOException {
+ public static StorageTypeProto getStorageTypeProto(StorageType type)
+ throws IOException {
StorageTypeProto storageTypeProto;
- switch (getStorageType()) {
+ switch (type) {
case SSD:
storageTypeProto = StorageTypeProto.SSD;
break;
@@ -145,7 +148,7 @@ public final class StorageLocationReport implements
* @return SCMStorageReport
* @throws IOException In case, the storage type specified is invalid.
*/
- public StorageReportProto getProtoBufMessage() throws IOException{
+ public StorageReportProto getProtoBufMessage() throws IOException {
StorageReportProto.Builder srb = StorageReportProto.newBuilder();
return srb.setStorageUuid(getId())
.setCapacity(getCapacity())
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerSpi.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerSpi.java
index 01f463c..d8dfefd 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerSpi.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerSpi.java
@@ -24,7 +24,8 @@ import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.protocol.proto
.StorageContainerDatanodeProtocolProtos.PipelineReport;
-
+import org.apache.hadoop.hdds.protocol.proto
+ .StorageContainerDatanodeProtocolProtos.MetadataStorageReportProto;
import java.io.IOException;
import java.util.Collection;
import java.util.List;
@@ -83,4 +84,13 @@ public interface XceiverServerSpi {
* @return list of report for each pipeline.
*/
List<PipelineReport> getPipelineReport();
+
+ /**
+ * Get storage report for the XceiverServer instance.
+ * @return list of report for each storage location.
+ */
+ default List<MetadataStorageReportProto> getStorageReport() throws
+ IOException {
+ return null;
+ }
}
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
index 3e6ac10..c751c5b 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
@@ -22,7 +22,6 @@ import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
-import java.util.Collections;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
@@ -30,6 +29,7 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
+import java.util.EnumMap;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
@@ -46,6 +46,7 @@ import org.apache.hadoop.hdds.HddsConfigKeys;
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandRequestProto;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.MetadataStorageReportProto;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ClosePipelineInfo;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineAction;
import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReport;
@@ -59,11 +60,14 @@ import org.apache.hadoop.hdds.utils.HddsServerUtil;
import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.OzoneConsts;
import org.apache.hadoop.ozone.container.common.impl.ContainerData;
+import org.apache.hadoop.ozone.container.common.impl.StorageLocationReport;
import org.apache.hadoop.ozone.container.common.interfaces.ContainerDispatcher;
import org.apache.hadoop.ozone.container.common.statemachine.StateContext;
import org.apache.hadoop.ozone.container.common.transport.server.XceiverServerSpi;
import org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet;
import org.apache.hadoop.ozone.container.ozoneimpl.ContainerController;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.server.datanode.StorageLocation;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
@@ -136,6 +140,11 @@ public final class XceiverServerRatis implements XceiverServerSpi {
// Timeout used while calling submitRequest directly.
private long requestTimeout;
+ /**
+ * Maintains a list of active volumes per StorageType.
+ */
+ private EnumMap<StorageType, List<String>> ratisVolumeMap;
+
private XceiverServerRatis(DatanodeDetails dd, int port,
ContainerDispatcher dispatcher, ContainerController containerController,
StateContext context, GrpcTlsConfig tlsConfig, ConfigurationSource conf)
@@ -163,6 +172,7 @@ public final class XceiverServerRatis implements XceiverServerSpi {
HddsConfigKeys.HDDS_DATANODE_RATIS_SERVER_REQUEST_TIMEOUT,
HddsConfigKeys.HDDS_DATANODE_RATIS_SERVER_REQUEST_TIMEOUT_DEFAULT,
TimeUnit.MILLISECONDS);
+ initializeRatisVolumeMap();
}
private ContainerStateMachine getStateMachine(RaftGroupId gid) {
@@ -213,9 +223,12 @@ public final class XceiverServerRatis implements XceiverServerSpi {
setNodeFailureTimeout(properties);
// Set the ratis storage directory
- String storageDir = HddsServerUtil.getOzoneDatanodeRatisDirectory(conf);
- RaftServerConfigKeys.setStorageDir(properties,
- Collections.singletonList(new File(storageDir)));
+ Collection<String> storageDirPaths =
+ HddsServerUtil.getOzoneDatanodeRatisDirectory(conf);
+ List<File> storageDirs= new ArrayList<>(storageDirPaths.size());
+ storageDirPaths.stream().forEach(d -> storageDirs.add(new File(d)));
+
+ RaftServerConfigKeys.setStorageDir(properties, storageDirs);
// For grpc set the maximum message size
GrpcConfigKeys.setMessageSizeMax(properties,
@@ -526,6 +539,43 @@ public final class XceiverServerRatis implements XceiverServerSpi {
}
}
+ private void initializeRatisVolumeMap() throws IOException {
+ ratisVolumeMap = new EnumMap<>(StorageType.class);
+ Collection<String> rawLocations = HddsServerUtil.
+ getOzoneDatanodeRatisDirectory(conf);
+
+ for (String locationString : rawLocations) {
+ try {
+ StorageLocation location = StorageLocation.parse(locationString);
+ StorageType type = location.getStorageType();
+ ratisVolumeMap.computeIfAbsent(type, k -> new ArrayList<String>(1));
+ ratisVolumeMap.get(location.getStorageType()).
+ add(location.getUri().getPath());
+
+ } catch (IOException e) {
+ LOG.error("Failed to parse the storage location: " +
+ locationString, e);
+ }
+ }
+ }
+
+ @Override
+ public List<MetadataStorageReportProto> getStorageReport()
+ throws IOException {
+ List<MetadataStorageReportProto> reportProto = new ArrayList<>();
+ for (StorageType storageType : ratisVolumeMap.keySet()) {
+ for (String path : ratisVolumeMap.get(storageType)) {
+ MetadataStorageReportProto.Builder builder = MetadataStorageReportProto.
+ newBuilder();
+ builder.setStorageLocation(path);
+ builder.setStorageType(StorageLocationReport.
+ getStorageTypeProto(storageType));
+ reportProto.add(builder.build());
+ }
+ }
+ return reportProto;
+ }
+
private RaftClientRequest createRaftClientRequest(
ContainerCommandRequestProto request, HddsProtos.PipelineID pipelineID,
RaftClientRequest.Type type) {
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java
index bc61811..b8c6067 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java
@@ -36,8 +36,6 @@ import java.util.concurrent.locks.ReentrantReadWriteLock;
import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.fs.SpaceUsageCheckFactory;
-import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
-import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.NodeReportProto;
import org.apache.hadoop.hdfs.server.datanode.StorageLocation;
import org.apache.hadoop.ozone.common.InconsistentStorageStateException;
import org.apache.hadoop.ozone.container.common.impl.StorageLocationReport;
@@ -487,7 +485,7 @@ public class MutableVolumeSet implements VolumeSet {
return ImmutableMap.copyOf(volumeStateMap);
}
- public StorageContainerDatanodeProtocolProtos.NodeReportProto getNodeReport()
+ public StorageLocationReport[] getStorageReport()
throws IOException {
boolean failed;
this.readLock();
@@ -540,11 +538,7 @@ public class MutableVolumeSet implements VolumeSet {
StorageLocationReport r = builder.build();
reports[counter++] = r;
}
- NodeReportProto.Builder nrb = NodeReportProto.newBuilder();
- for (int i = 0; i < reports.length; i++) {
- nrb.addStorageReport(reports[i].getProtoBufMessage());
- }
- return nrb.build();
+ return reports;
} finally {
this.readUnlock();
}
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
index bbbec25..62fd5a4 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient
import org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics;
import org.apache.hadoop.ozone.container.common.impl.ContainerSet;
import org.apache.hadoop.ozone.container.common.impl.HddsDispatcher;
+import org.apache.hadoop.ozone.container.common.impl.StorageLocationReport;
import org.apache.hadoop.ozone.container.common.interfaces.ContainerDispatcher;
import org.apache.hadoop.ozone.container.common.interfaces.Handler;
import org.apache.hadoop.ozone.container.common.statemachine.StateContext;
@@ -294,8 +295,21 @@ public class OzoneContainer {
* Returns node report of container storage usage.
*/
public StorageContainerDatanodeProtocolProtos.NodeReportProto getNodeReport()
- throws IOException {
- return volumeSet.getNodeReport();
+ throws IOException {
+ StorageLocationReport[] reports = volumeSet.getStorageReport();
+ StorageContainerDatanodeProtocolProtos.NodeReportProto.Builder nrb
+ = StorageContainerDatanodeProtocolProtos.
+ NodeReportProto.newBuilder();
+ for (int i = 0; i < reports.length; i++) {
+ nrb.addStorageReport(reports[i].getProtoBufMessage());
+ }
+ List<StorageContainerDatanodeProtocolProtos.
+ MetadataStorageReportProto> metadataReport =
+ writeChannel.getStorageReport();
+ if (metadataReport != null) {
+ nrb.addAllMetadataStorageReport(metadataReport);
+ }
+ return nrb.build();
}
@VisibleForTesting
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
index 1056a0d..2bb52f6 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
@@ -29,6 +29,7 @@ import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.OzoneConsts;
import org.apache.hadoop.ozone.container.common.helpers.BlockData;
import org.apache.hadoop.ozone.container.common.helpers.ChunkInfo;
@@ -45,10 +46,7 @@ import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer;
import org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerData;
import org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils;
import org.apache.hadoop.test.LambdaTestUtils;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
+import org.junit.*;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
@@ -170,6 +168,51 @@ public class TestOzoneContainer {
}
@Test
+ public void testBuildNodeReport() throws Exception {
+ String path = folder.getRoot()
+ .getAbsolutePath();
+ conf.set(OzoneConfigKeys.DFS_CONTAINER_RATIS_DATANODE_STORAGE_DIR,
+ String.join(",",
+ path + "/ratis1", path + "/ratis2", path + "ratis3"));
+ DatanodeStateMachine stateMachine = Mockito.mock(
+ DatanodeStateMachine.class);
+ StateContext context = Mockito.mock(StateContext.class);
+ Mockito.when(stateMachine.getDatanodeDetails()).thenReturn(datanodeDetails);
+ Mockito.when(context.getParent()).thenReturn(stateMachine);
+ // When OzoneContainer is started, the containers from disk should be
+ // loaded into the containerSet.
+ // Also expected to initialize committed space for each volume.
+ OzoneContainer ozoneContainer = new
+ OzoneContainer(datanodeDetails, conf, context, null);
+ Assert.assertEquals(volumeSet.getVolumesList().size(),
+ ozoneContainer.getNodeReport().getStorageReportList().size());
+ Assert.assertEquals(3,
+ ozoneContainer.getNodeReport().getMetadataStorageReportList()
+ .size());
+
+ }
+
+ @Test
+ public void testBuildNodeReportWithDefaultRatisLogDir() throws Exception {
+ DatanodeStateMachine stateMachine = Mockito.mock(
+ DatanodeStateMachine.class);
+ StateContext context = Mockito.mock(StateContext.class);
+ Mockito.when(stateMachine.getDatanodeDetails()).thenReturn(datanodeDetails);
+ Mockito.when(context.getParent()).thenReturn(stateMachine);
+ // When OzoneContainer is started, the containers from disk should be
+ // loaded into the containerSet.
+ // Also expected to initialize committed space for each volume.
+ OzoneContainer ozoneContainer = new
+ OzoneContainer(datanodeDetails, conf, context, null);
+ Assert.assertEquals(volumeSet.getVolumesList().size(),
+ ozoneContainer.getNodeReport().getStorageReportList().size());
+ Assert.assertEquals(1,
+ ozoneContainer.getNodeReport().getMetadataStorageReportList()
+ .size());
+ }
+
+
+ @Test
public void testContainerCreateDiskFull() throws Exception {
long containerSize = (long) StorageUnit.MB.toBytes(100);
diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java
index ee51142..8e7f326 100644
--- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java
+++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java
@@ -20,6 +20,8 @@ package org.apache.hadoop.hdds.utils;
import java.io.File;
import java.io.IOException;
import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Collection;
import java.util.Optional;
import java.util.OptionalInt;
import java.util.concurrent.TimeUnit;
@@ -51,7 +53,6 @@ import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.apache.hadoop.ozone.OzoneConsts;
import org.apache.hadoop.security.UserGroupInformation;
-import com.google.common.base.Strings;
import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_HEARTBEAT_INTERVAL;
import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_HEARTBEAT_INTERVAL_DEFAULT;
import static org.apache.hadoop.hdds.HddsUtils.getHostNameFromConfigKeys;
@@ -344,15 +345,16 @@ public final class HddsServerUtil {
OzoneConfigKeys.DFS_CONTAINER_IPC_PORT_DEFAULT);
}
- public static String getOzoneDatanodeRatisDirectory(
+ public static Collection<String> getOzoneDatanodeRatisDirectory(
ConfigurationSource conf) {
- String storageDir = conf.get(
- OzoneConfigKeys.DFS_CONTAINER_RATIS_DATANODE_STORAGE_DIR);
+ Collection<String> rawLocations = conf.getTrimmedStringCollection(
+ OzoneConfigKeys.DFS_CONTAINER_RATIS_DATANODE_STORAGE_DIR);
- if (Strings.isNullOrEmpty(storageDir)) {
- storageDir = ServerUtils.getDefaultRatisDirectory(conf);
+ if (rawLocations.isEmpty()) {
+ rawLocations = new ArrayList<>(1);
+ rawLocations.add(ServerUtils.getDefaultRatisDirectory(conf));
}
- return storageDir;
+ return rawLocations;
}
/**
diff --git a/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto b/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto
index 191a81a..00c8fdb 100644
--- a/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto
+++ b/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto
@@ -146,6 +146,7 @@ message SCMNodeAddressList {
*/
message NodeReportProto {
repeated StorageReportProto storageReport = 1;
+ repeated MetadataStorageReportProto metadataStorageReport = 2;
}
message StorageReportProto {
@@ -158,6 +159,11 @@ message StorageReportProto {
optional bool failed = 7 [default = false];
}
+message MetadataStorageReportProto {
+ required string storageLocation = 1;
+ optional StorageTypeProto storageType = 2 [default = DISK];
+}
+
/**
* Types of recognized storage media.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 15/39: HDDS-2768. security/SecuringS3.md (#1188)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 880adec2e6c4c808495fe465d7d0306b241f0be1
Author: Xiang Zhang <xi...@gmail.com>
AuthorDate: Wed Jul 15 14:37:47 2020 +0800
HDDS-2768. security/SecuringS3.md (#1188)
(cherry picked from commit 10686014b98a01c30f54614e172f57ab99c48c5b)
---
hadoop-hdds/docs/content/security/SecuringS3.md | 6 +--
hadoop-hdds/docs/content/security/SecuringS3.zh.md | 50 ++++++++++++++++++++++
2 files changed, 53 insertions(+), 3 deletions(-)
diff --git a/hadoop-hdds/docs/content/security/SecuringS3.md b/hadoop-hdds/docs/content/security/SecuringS3.md
index 1cb0c80..578aa33 100644
--- a/hadoop-hdds/docs/content/security/SecuringS3.md
+++ b/hadoop-hdds/docs/content/security/SecuringS3.md
@@ -38,14 +38,14 @@ The user needs to `kinit` first and once they have authenticated via kerberos
```bash
ozone s3 getsecret
```
-This command will talk to ozone, validate the user via kerberos and generate
+This command will talk to ozone, validate the user via Kerberos and generate
the AWS credentials. The values will be printed out on the screen. You can
set these values up in your _.aws_ file for automatic access while working
against Ozone S3 buckets.
<div class="alert alert-danger" role="alert">
- Please note: These S3 crediantials are like your kerberos passswords
- that give compelete access to your buckets.
+ Please note: These S3 credentials are like your Kerberos passwords
+ that give complete access to your buckets.
</div>
diff --git a/hadoop-hdds/docs/content/security/SecuringS3.zh.md b/hadoop-hdds/docs/content/security/SecuringS3.zh.md
new file mode 100644
index 0000000..b7d52b7
--- /dev/null
+++ b/hadoop-hdds/docs/content/security/SecuringS3.zh.md
@@ -0,0 +1,50 @@
+---
+title: "安全化 S3"
+date: "2019-April-03"
+summary: Ozone 支持 S3 协议，并使用 AWS Signature Version 4 protocol which allows a seamless S3
+ experience.
+weight: 4
+icon: cloud
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+用户需要由 AWS 网站生成的 AWS access key ID 和 AWS secret 来访问 AWS S3 的桶，当你使用 Ozone 的 S3 协议时，你也需要同样的 AWS access key 和 secret。
+
+在 Ozone 中，用户可以直接下载 access key。用户需要先执行 `kinit` 命令进行 Kerberos 认证，认证通过后就可以下载 S3 access key 和 secret。和 AWS S3 一样，access key 和 secret 具有 S3 桶的全部权限，用户需要保管好 key 和 secret。
+
+* S3 客户端可以从 OM 获取 access key id 和 secret。
+
+```bash
+ozone s3 getsecret
+```
+这条命令会与 Ozone 进行通信，对用户进行 Kerberos 认证并生成 AWS 凭据，结果会直接打印在屏幕上，你可以将其配置在 _.aws._ 文件中，这样可以在操作 Ozone S3 桶时自动进行认证。
+
+<div class="alert alert-danger" role="alert">
+请注意：这些 S3 凭据和你的 Kerberos 密码一样，具有你所有桶的完全访问权限。
+</div>
+
+
+* 在 aws 配置中添加上述凭据：
+
+```bash
+aws configure set default.s3.signature_version s3v4
+aws configure set aws_access_key_id ${accessId}
+aws configure set aws_secret_access_key ${secret}
+aws configure set region us-west-1
+```
+关于通过命令行和 S3 API 使用 S3，请参考 AWS S3 的文档。
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 02/39: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen… (#1162)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 4aa6087050c6d3324f7808098d3e15c6acac46e9
Author: Sammi Chen <sa...@apache.org>
AuthorDate: Mon Jul 13 14:38:34 2020 +0800
HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen… (#1162)
(cherry picked from commit 59771684d25f3f75ce4d97bc877335e9e5897e03)
---
.../apache/hadoop/hdds/scm/container/ReplicationManager.java | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
index 617e231..7a25068 100644
--- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
+++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
@@ -512,7 +512,7 @@ public class ReplicationManager
*/
private void handleUnderReplicatedContainer(final ContainerInfo container,
final Set<ContainerReplica> replicas) {
- LOG.debug("Handling underreplicated container: {}",
+ LOG.debug("Handling under-replicated container: {}",
container.getContainerID());
try {
final ContainerID id = container.containerID();
@@ -543,14 +543,18 @@ public class ReplicationManager
List<DatanodeDetails> targetReplicas = new ArrayList<>(source);
// Then add any pending additions
targetReplicas.addAll(replicationInFlight);
-
- int delta = replicationFactor - getReplicaCount(id, replicas);
final ContainerPlacementStatus placementStatus =
containerPlacement.validateContainerPlacement(
targetReplicas, replicationFactor);
+ int delta = replicationFactor - getReplicaCount(id, replicas);
final int misRepDelta = placementStatus.misReplicationCount();
final int replicasNeeded
= delta < misRepDelta ? misRepDelta : delta;
+ if (replicasNeeded <= 0) {
+ LOG.debug("Container {} meets replication requirement with " +
+ "inflight replicas", id);
+ return;
+ }
final List<DatanodeDetails> excludeList = replicas.stream()
.map(ContainerReplica::getDatanodeDetails)
@@ -611,7 +615,7 @@ public class ReplicationManager
final ContainerID id = container.containerID();
final int replicationFactor = container.getReplicationFactor().getNumber();
- // Dont consider inflight replication while calculating excess here.
+ // Don't consider inflight replication while calculating excess here.
int excess = replicas.size() - replicationFactor -
inflightDeletion.getOrDefault(id, Collections.emptyList()).size();
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 21/39: HDDS-3958. Intermittent failure in Recon
acceptance test due to mixed stdout and stderr (#1200)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 813a4c4e004dd22e32357d451028b5c547ea9a97
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Thu Jul 16 03:40:50 2020 +0200
HDDS-3958. Intermittent failure in Recon acceptance test due to mixed stdout and stderr (#1200)
(cherry picked from commit d3e54fb6293a44ea9f2ffdfa2a534d3b643ab567)
---
.../dist/src/main/smoketest/recon/recon-api.robot | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot b/hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot
index 7073849..ada2dd8 100644
--- a/hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot
@@ -28,10 +28,10 @@ ${API_ENDPOINT_URL} http://recon:9888/api/v1
*** Keywords ***
Check if Recon picks up container from OM
Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit HTTP user
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/containers
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/containers
Should contain ${result} \"ContainerID\":1
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/utilization/fileCount
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/utilization/fileCount
Should contain ${result} \"fileSize\":2048,\"count\":10
*** Test Cases ***
@@ -43,13 +43,13 @@ Check if Recon picks up OM data
Wait Until Keyword Succeeds 90sec 10sec Check if Recon picks up container from OM
Check if Recon picks up DN heartbeats
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/datanodes
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/datanodes
Should contain ${result} datanodes
Should contain ${result} datanode_1
Should contain ${result} datanode_2
Should contain ${result} datanode_3
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/pipelines
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/pipelines
Should contain ${result} pipelines
Should contain ${result} RATIS
Should contain ${result} OPEN
@@ -57,15 +57,15 @@ Check if Recon picks up DN heartbeats
Should contain ${result} datanode_2
Should contain ${result} datanode_3
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/clusterState
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/clusterState
Should contain ${result} \"totalDatanodes\":3
Should contain ${result} \"healthyDatanodes\":3
Should contain ${result} \"pipelines\":4
- ${result} = Execute curl --negotiate -u : -v ${API_ENDPOINT_URL}/containers/1/replicaHistory
+ ${result} = Execute curl --negotiate -u : -LSs ${API_ENDPOINT_URL}/containers/1/replicaHistory
Should contain ${result} \"containerId\":1
Check if Recon Web UI is up
Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit HTTP user
- ${result} = Execute curl --negotiate -u : -v ${ENDPOINT_URL}
- Should contain ${result} Ozone Recon
\ No newline at end of file
+ ${result} = Execute curl --negotiate -u : -LSs ${ENDPOINT_URL}
+ Should contain ${result} Ozone Recon
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 35/39: HDDS-3813. Upgrade Ratis third-party,
too (#1229)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit a86afab65a01647749fe2792e0370d104629bc71
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Tue Jul 21 14:07:36 2020 +0200
HDDS-3813. Upgrade Ratis third-party, too (#1229)
(cherry picked from commit 798e00c97d228212c45be164b756e8250400eacf)
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index bcdaf69..6a9f864 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,7 +82,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs
<ratis.version>1.0.0</ratis.version>
<!-- Apache Ratis thirdparty version -->
- <ratis.thirdparty.version>0.4.0</ratis.thirdparty.version>
+ <ratis.thirdparty.version>0.5.0</ratis.thirdparty.version>
<distMgmtSnapshotsId>apache.snapshots.https</distMgmtSnapshotsId>
<distMgmtSnapshotsName>Apache Development Snapshot Repository</distMgmtSnapshotsName>
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 34/39: Update ratis to 1.0.0 (#1222)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit f58883ba8c036f1385901afa94fa22667669557b
Author: Lokesh Jain <lj...@apache.org>
AuthorDate: Tue Jul 21 14:47:46 2020 +0530
Update ratis to 1.0.0 (#1222)
(cherry picked from commit fb2649e146f068318f628867f7ef1326906f6d92)
---
.../common/transport/server/ratis/ContainerStateMachine.java | 12 ++++++------
pom.xml | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
index b1c8370..840d87c 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
@@ -103,7 +103,7 @@ import org.slf4j.LoggerFactory;
* processed in 2 phases. The 2 phases are divided in
* {@link #startTransaction(RaftClientRequest)}, in the first phase the user
* data is written directly into the state machine via
- * {@link #writeStateMachineData} and in the second phase the
+ * {@link #write} and in the second phase the
* transaction is committed via {@link #applyTransaction(TransactionContext)}
*
* For the requests with no stateMachine data, the transaction is directly
@@ -115,7 +115,7 @@ import org.slf4j.LoggerFactory;
* the write chunk operation will fail otherwise as the container still hasn't
* been created. Hence the create container operation has been split in the
* {@link #startTransaction(RaftClientRequest)}, this will help in synchronizing
- * the calls in {@link #writeStateMachineData}
+ * the calls in {@link #write}
*
* 2) Write chunk commit operation is executed after write chunk state machine
* operation. This will ensure that commit operation is sync'd with the state
@@ -517,7 +517,7 @@ public class ContainerStateMachine extends BaseStateMachine {
* and also with applyTransaction.
*/
@Override
- public CompletableFuture<Message> writeStateMachineData(LogEntryProto entry) {
+ public CompletableFuture<Message> write(LogEntryProto entry) {
try {
metrics.incNumWriteStateMachineOps();
long writeStateMachineStartTime = Time.monotonicNowNanos();
@@ -618,7 +618,7 @@ public class ContainerStateMachine extends BaseStateMachine {
* @return Combined future of all writeChunks till the log index given.
*/
@Override
- public CompletableFuture<Void> flushStateMachineData(long index) {
+ public CompletableFuture<Void> flush(long index) {
List<CompletableFuture<ContainerCommandResponseProto>> futureList =
writeChunkFutureMap.entrySet().stream().filter(x -> x.getKey() <= index)
.map(Map.Entry::getValue).collect(Collectors.toList());
@@ -632,7 +632,7 @@ public class ContainerStateMachine extends BaseStateMachine {
* evicted.
*/
@Override
- public CompletableFuture<ByteString> readStateMachineData(
+ public CompletableFuture<ByteString> read(
LogEntryProto entry) {
StateMachineLogEntryProto smLogEntryProto = entry.getStateMachineLogEntry();
metrics.incNumReadStateMachineOps();
@@ -833,7 +833,7 @@ public class ContainerStateMachine extends BaseStateMachine {
}
@Override
- public CompletableFuture<Void> truncateStateMachineData(long index) {
+ public CompletableFuture<Void> truncate(long index) {
stateMachineDataCache.removeIf(k -> k >= index);
return CompletableFuture.completedFuture(null);
}
diff --git a/pom.xml b/pom.xml
index 40f2f58..bcdaf69 100644
--- a/pom.xml
+++ b/pom.xml
@@ -79,7 +79,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs
<declared.ozone.version>${ozone.version}</declared.ozone.version>
<!-- Apache Ratis version -->
- <ratis.version>0.6.0-6ab75ae-SNAPSHOT</ratis.version>
+ <ratis.version>1.0.0</ratis.version>
<!-- Apache Ratis thirdparty version -->
<ratis.thirdparty.version>0.4.0</ratis.thirdparty.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 09/39: HDDS-3062. Fix
TestOzoneRpcClientAbstract.testListVolume - addendum for checkstyle
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 69bb0eea8aa61d3245272bc6527f240541133e61
Author: Doroszlai, Attila <ad...@apache.org>
AuthorDate: Tue Jul 14 15:30:01 2020 +0200
HDDS-3062. Fix TestOzoneRpcClientAbstract.testListVolume - addendum for checkstyle
(cherry picked from commit 415d4aaff20e23214ddccc6578a61854ff12db47)
---
.../org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java | 1 -
1 file changed, 1 deletion(-)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
index f0ffee3..3b90815 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
@@ -118,7 +118,6 @@ import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
-import org.junit.Ignore;
import org.junit.Test;
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 10/39: HDDS-3957. Fix mixed use of Longs.toByteArray
and Ints.fromByteArray (#1199)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit b3139e80b3d8565100ca6c88357b8e4010e89776
Author: runzhiwang <51...@users.noreply.github.com>
AuthorDate: Wed Jul 15 00:20:16 2020 +0800
HDDS-3957. Fix mixed use of Longs.toByteArray and Ints.fromByteArray (#1199)
(cherry picked from commit 6d979e49190a48fc1c43edc0ff2b410eae09381d)
---
.../impl/TopNOrderedContainerDeletionChoosingPolicy.java | 2 +-
.../ozone/container/keyvalue/KeyValueContainerData.java | 14 +++++++-------
.../container/keyvalue/helpers/KeyValueContainerUtil.java | 3 +--
.../ozone/container/common/TestBlockDeletingService.java | 7 ++++---
4 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java
index 41fc267..2cee75c 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/TopNOrderedContainerDeletionChoosingPolicy.java
@@ -45,7 +45,7 @@ public class TopNOrderedContainerDeletionChoosingPolicy
private static final Comparator<KeyValueContainerData>
KEY_VALUE_CONTAINER_DATA_COMPARATOR = (KeyValueContainerData c1,
KeyValueContainerData c2) ->
- Integer.compare(c2.getNumPendingDeletionBlocks(),
+ Long.compare(c2.getNumPendingDeletionBlocks(),
c1.getNumPendingDeletionBlocks());
@Override
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
index 5698d72..68f01fb 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerData.java
@@ -38,7 +38,7 @@ import org.yaml.snakeyaml.nodes.Tag;
import java.io.File;
import java.util.List;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
import static java.lang.Math.max;
import static org.apache.hadoop.ozone.OzoneConsts.CONTAINER_DB_TYPE_ROCKSDB;
@@ -73,7 +73,7 @@ public class KeyValueContainerData extends ContainerData {
/**
* Number of pending deletion blocks in KeyValueContainer.
*/
- private final AtomicInteger numPendingDeletionBlocks;
+ private final AtomicLong numPendingDeletionBlocks;
private long deleteTransactionId;
@@ -98,7 +98,7 @@ public class KeyValueContainerData extends ContainerData {
long size, String originPipelineId, String originNodeId) {
super(ContainerProtos.ContainerType.KeyValueContainer, id, layOutVersion,
size, originPipelineId, originNodeId);
- this.numPendingDeletionBlocks = new AtomicInteger(0);
+ this.numPendingDeletionBlocks = new AtomicLong(0);
this.deleteTransactionId = 0;
}
@@ -106,7 +106,7 @@ public class KeyValueContainerData extends ContainerData {
super(source);
Preconditions.checkArgument(source.getContainerType()
== ContainerProtos.ContainerType.KeyValueContainer);
- this.numPendingDeletionBlocks = new AtomicInteger(0);
+ this.numPendingDeletionBlocks = new AtomicLong(0);
this.deleteTransactionId = 0;
}
@@ -188,7 +188,7 @@ public class KeyValueContainerData extends ContainerData {
*
* @param numBlocks increment number
*/
- public void incrPendingDeletionBlocks(int numBlocks) {
+ public void incrPendingDeletionBlocks(long numBlocks) {
this.numPendingDeletionBlocks.addAndGet(numBlocks);
}
@@ -197,14 +197,14 @@ public class KeyValueContainerData extends ContainerData {
*
* @param numBlocks decrement number
*/
- public void decrPendingDeletionBlocks(int numBlocks) {
+ public void decrPendingDeletionBlocks(long numBlocks) {
this.numPendingDeletionBlocks.addAndGet(-1 * numBlocks);
}
/**
* Get the number of pending deletion blocks.
*/
- public int getNumPendingDeletionBlocks() {
+ public long getNumPendingDeletionBlocks() {
return this.numPendingDeletionBlocks.get();
}
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
index 2141bed..d5fea23 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
@@ -23,7 +23,6 @@ import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
-import com.google.common.primitives.Ints;
import com.google.common.primitives.Longs;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
@@ -171,7 +170,7 @@ public final class KeyValueContainerUtil {
containerDB.getStore().get(DB_PENDING_DELETE_BLOCK_COUNT_KEY);
if (pendingDeleteBlockCount != null) {
kvContainerData.incrPendingDeletionBlocks(
- Ints.fromByteArray(pendingDeleteBlockCount));
+ Longs.fromByteArray(pendingDeleteBlockCount));
} else {
// Set pending deleted block count.
MetadataKeyFilters.KeyPrefixFilter filter =
diff --git a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
index b8843de..ba6999d 100644
--- a/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
+++ b/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
@@ -27,7 +27,6 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import com.google.common.collect.Lists;
-import com.google.common.primitives.Ints;
import com.google.common.primitives.Longs;
import org.apache.commons.io.FileUtils;
import org.apache.hadoop.hdds.StringUtils;
@@ -180,7 +179,7 @@ public class TestBlockDeletingService {
metadata.getStore().put(OzoneConsts.DB_CONTAINER_BYTES_USED_KEY,
Longs.toByteArray(blockLength * numOfBlocksPerContainer));
metadata.getStore().put(DB_PENDING_DELETE_BLOCK_COUNT_KEY,
- Ints.toByteArray(numOfBlocksPerContainer));
+ Longs.toByteArray(numOfBlocksPerContainer));
}
}
}
@@ -251,6 +250,8 @@ public class TestBlockDeletingService {
// Ensure there are 3 blocks under deletion and 0 deleted blocks
Assert.assertEquals(3, getUnderDeletionBlocksCount(meta));
+ Assert.assertEquals(3, Longs.fromByteArray(
+ meta.getStore().get(DB_PENDING_DELETE_BLOCK_COUNT_KEY)));
Assert.assertEquals(0, getDeletedBlocksCount(meta));
// An interval will delete 1 * 2 blocks
@@ -269,7 +270,7 @@ public class TestBlockDeletingService {
// Check finally DB counters.
// Not checking bytes used, as handler is a mock call.
- Assert.assertEquals(0, Ints.fromByteArray(
+ Assert.assertEquals(0, Longs.fromByteArray(
meta.getStore().get(DB_PENDING_DELETE_BLOCK_COUNT_KEY)));
Assert.assertEquals(0, Longs.fromByteArray(
meta.getStore().get(DB_BLOCK_COUNT_KEY)));
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 17/39: HDDS-3798. Display more accurate timestamp in
recon Web (#1201)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 2a66d324cdaf87078892c89d8815cff074d8825c
Author: HuangTao <th...@163.com>
AuthorDate: Wed Jul 15 23:24:02 2020 +0800
HDDS-3798. Display more accurate timestamp in recon Web (#1201)
(cherry picked from commit da49ca6891cc7a2f6ed44aacf8ec51331ad969f4)
---
.../src/components/autoReloadPanel/autoReloadPanel.tsx | 4 ++--
.../webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx | 4 ++--
.../webapps/recon/ozone-recon-web/src/views/pipelines/pipelines.tsx | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/autoReloadPanel/autoReloadPanel.tsx b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/autoReloadPanel/autoReloadPanel.tsx
index fee9a88..773828d 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/autoReloadPanel/autoReloadPanel.tsx
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/autoReloadPanel/autoReloadPanel.tsx
@@ -42,9 +42,9 @@ class AutoReloadPanel extends React.Component<IAutoReloadPanelProps> {
const lastUpdatedText = lastUpdated === 0 ? 'NA' :
(
<Tooltip
- placement='bottom' title={moment(lastUpdated).format('lll')}
+ placement='bottom' title={moment(lastUpdated).format('ll LTS')}
>
- {moment(lastUpdated).format('LT')}
+ {moment(lastUpdated).format('LTS')}
</Tooltip>
);
return (
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
index 877ebf9..fada1f4 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/datanodes/datanodes.tsx
@@ -135,7 +135,7 @@ const COLUMNS = [
isVisible: true,
sorter: (a: IDatanode, b: IDatanode) => a.lastHeartbeat - b.lastHeartbeat,
render: (heartbeat: number) => {
- return heartbeat > 0 ? moment(heartbeat).format('lll') : 'NA';
+ return heartbeat > 0 ? moment(heartbeat).format('ll LTS') : 'NA';
}
},
{
@@ -197,7 +197,7 @@ const COLUMNS = [
isVisible: false,
sorter: (a: IDatanode, b: IDatanode) => a.setupTime - b.setupTime,
render: (uptime: number) => {
- return uptime > 0 ? moment(uptime).format('lll') : 'NA';
+ return uptime > 0 ? moment(uptime).format('ll LTS') : 'NA';
}
}
];
diff --git a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/pipelines/pipelines.tsx b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/pipelines/pipelines.tsx
index 15b6858..b898818 100644
--- a/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/pipelines/pipelines.tsx
+++ b/hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/pipelines/pipelines.tsx
@@ -114,7 +114,7 @@ const COLUMNS = [
dataIndex: 'lastLeaderElection',
key: 'lastLeaderElection',
render: (lastLeaderElection: number) => lastLeaderElection > 0 ?
- moment(lastLeaderElection).format('lll') : 'NA',
+ moment(lastLeaderElection).format('ll LTS') : 'NA',
sorter: (a: IPipelineResponse, b: IPipelineResponse) => a.lastLeaderElection - b.lastLeaderElection
},
{
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 12/39: HDDS-3060. /retest github comment does not
work (#1137)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 99e7b7a6c4c02265971c3afb8c24ac196b802de8
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Tue Jul 14 20:56:01 2020 +0200
HDDS-3060. /retest github comment does not work (#1137)
(cherry picked from commit 8573e5b72b5aec9097f485f3f680043f6ffe0b53)
---
.github/comment-commands/retest.sh | 66 +++++++++++++++++++++++++++++++-------
1 file changed, 54 insertions(+), 12 deletions(-)
diff --git a/.github/comment-commands/retest.sh b/.github/comment-commands/retest.sh
index b9e0632..2214264 100755
--- a/.github/comment-commands/retest.sh
+++ b/.github/comment-commands/retest.sh
@@ -14,20 +14,62 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-#doc: add new empty commit to trigger new CI build
-set +x #GITHUB_TOKEN
+#doc: provide help on how to trigger new CI build
-PR_URL=$(jq -r '.issue.pull_request.url' "$GITHUB_EVENT_PATH")
-read -r REPO_URL BRANCH <<<"$(curl "$PR_URL" | jq -r '.head.repo.clone_url + " " + .head.ref' | sed "s/github.com/$GITHUB_ACTOR:$GITHUB_TOKEN@github.com/g")"
+# posting a new commit from this script does not trigger CI checks
+# https://help.github.com/en/actions/reference/events-that-trigger-workflows#triggering-new-workflows-using-a-personal-access-token
-git fetch "$REPO_URL" "$BRANCH"
-git checkout FETCH_HEAD
+set -eu
+
+code='```'
+
+pr_url="$(jq -r '.issue.pull_request.url' "${GITHUB_EVENT_PATH}")"
+commenter="$(jq -r '.comment.user.login' "${GITHUB_EVENT_PATH}")"
+assoc="$(jq -r '.comment.author_association' "${GITHUB_EVENT_PATH}")"
-export GIT_COMMITTER_EMAIL="noreply@github.com"
-export GIT_COMMITTER_NAME="GitHub actions"
+curl -LSs "${pr_url}" -o pull.tmp
+source_repo="$(jq -r '.head.repo.ssh_url' pull.tmp)"
+branch="$(jq -r '.head.ref' pull.tmp)"
+pr_owner="$(jq -r '.head.user.login' pull.tmp)"
+maintainer_can_modify="$(jq -r '.maintainer_can_modify' pull.tmp)"
-export GIT_AUTHOR_EMAIL="noreply@github.com"
-export GIT_AUTHOR_NAME="GitHub actions"
+# PR owner
+# =>
+# has local branch, can simply push
+if [[ "${commenter}" == "${pr_owner}" ]]; then
+ cat <<-EOF
+To re-run CI checks, please follow these steps with the source branch checked out:
+${code}
+git commit --allow-empty -m 'trigger new CI check'
+git push
+${code}
+EOF
+
+# member AND modification allowed by PR author
+# OR
+# repo owner
+# =>
+# include steps to fetch branch
+elif [[ "${maintainer_can_modify}" == "true" ]] && [[ "${assoc}" == "MEMBER" ]] || [[ "${assoc}" == "OWNER" ]]; then
+ cat <<-EOF
+To re-run CI checks, please follow these steps:
+${code}
+git fetch "${source_repo}" "${branch}"
+git checkout FETCH_HEAD
+git commit --allow-empty -m 'trigger new CI check'
+git push "${source_repo}" HEAD:"${branch}"
+${code}
+EOF
-git commit --allow-empty -m "empty commit to retest build" > /dev/null
-git push $REPO_URL HEAD:$BRANCH
+# other folks
+# =>
+# ping author
+else
+ cat <<-EOF
+@${pr_owner} please trigger new CI check by following these steps:
+${code}
+git commit --allow-empty -m 'trigger new CI check'
+git push
+${code}
+EOF
+fi
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 20/39: HDDS-2767. security/SecuringTDE.md (#1184)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 0c8f2c839f83dbe17ae31324ea2e1fbb75b1c525
Author: Xiang Zhang <xi...@gmail.com>
AuthorDate: Thu Jul 16 09:03:35 2020 +0800
HDDS-2767. security/SecuringTDE.md (#1184)
(cherry picked from commit 22d03f657a1680875a2e022b45e94c87e080188b)
---
hadoop-hdds/docs/content/security/SecuringTDE.md | 4 +-
.../docs/content/security/SecuringTDE.zh.md | 52 ++++++++++++++++++++++
2 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/hadoop-hdds/docs/content/security/SecuringTDE.md b/hadoop-hdds/docs/content/security/SecuringTDE.md
index 3e8f2d1..0e21b62 100644
--- a/hadoop-hdds/docs/content/security/SecuringTDE.md
+++ b/hadoop-hdds/docs/content/security/SecuringTDE.md
@@ -1,7 +1,7 @@
---
title: "Transparent Data Encryption"
date: "2019-April-03"
-summary: TDE allows data on the disks to be encrypted-at-rest and automatically decrypted during access. You can enable this per key or per bucket.
+summary: TDE allows data on the disks to be encrypted-at-rest and automatically decrypted during access.
weight: 3
icon: lock
---
@@ -28,7 +28,7 @@ when a bucket is created.
### Setting up the Key Management Server
-To use TDE, clients must setup a Key Management Server and provide that URI to
+To use TDE, admin must setup a Key Management Server and provide that URI to
Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this
configuration can be provided via *hdfs-site.xml*.
diff --git a/hadoop-hdds/docs/content/security/SecuringTDE.zh.md b/hadoop-hdds/docs/content/security/SecuringTDE.zh.md
new file mode 100644
index 0000000..35baaa0
--- /dev/null
+++ b/hadoop-hdds/docs/content/security/SecuringTDE.zh.md
@@ -0,0 +1,52 @@
+---
+title: "透明数据加密"
+date: "2019-April-03"
+summary: 透明数据加密（Transparent Data Encryption，TDE）以密文形式在磁盘上保存数据，但可以在用户访问的时候自动进行解密。
+weight: 3
+icon: lock
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Ozone TDE 的配置和使用和 HDFS TDE 十分相似，主要的区别是，Ozone 中桶级别的 TDE 必须在创建桶时启用。
+
+### 配置密钥管理服务器
+
+使用 TDE 之前，管理员必须要提前配置密钥管理服务 KMS，并且把 KMS 的 URI 通过 core-site.xml 提供给 Ozone。
+
+参数名 | 值
+-----------------------------------|-----------------------------------------
+hadoop.security.key.provider.path | KMS uri. <br> 比如 kms://http@kms-host:9600/kms
+
+### 使用 TDE
+如果你的集群已经配置好了 TDE，那么你只需要创建加密密钥并启用桶加密即可。
+
+创建加密密钥的方法为：
+ * 使用 hadoop key 命令创建桶加密密钥，和 HDFS 加密区域的使用方法类似。
+
+ ```bash
+ hadoop key create encKey
+ ```
+ 上面这个命令会创建一个用于保护桶数据的密钥。创建完成之后，你可以告诉 Ozone 在读写某个桶中的数据时使用这个密钥。
+
+ * 将加密密钥分配给桶
+
+ ```bash
+ ozone sh bucket create -k encKey /vol/encryptedBucket
+ ```
+
+这条命令执行后，所以写往 _encryptedBucket_ 的数据都会用 encKey 进行加密，当读取里面的数据时，客户端通过 KMS 获取密钥进行解密。换句话说，Ozone 中存储的数据一直是加密的，但用户和客户端对此完全无感知。
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 31/39: HDDS-3806. Support recognize aws v2
Authorization header. (#1098)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 8b346ede3b2687ff81624653587e3bcb5a0773c4
Author: Sammi Chen <sa...@apache.org>
AuthorDate: Mon Jul 20 15:27:10 2020 +0800
HDDS-3806. Support recognize aws v2 Authorization header. (#1098)
(cherry picked from commit c64d226243541f2bc0d89ab515156f84fa0cfd7e)
---
...reProcessor.java => AWSSignatureProcessor.java} | 35 ++++++++++++-----
.../hadoop/ozone/s3/OzoneClientProducer.java | 8 ++--
.../ozone/s3/header/AuthorizationHeaderV2.java | 2 +-
.../ozone/s3/header/AuthorizationHeaderV4.java | 4 +-
...ocessor.java => TestAWSSignatureProcessor.java} | 44 ++++++++++++++++++++--
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 2 +-
.../ozone/s3/header/TestAuthorizationHeaderV4.java | 2 +-
7 files changed, 75 insertions(+), 22 deletions(-)
diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSSignatureProcessor.java
similarity index 92%
rename from hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java
rename to hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSSignatureProcessor.java
index 1ff1a72..0cb82fb 100644
--- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSV4SignatureProcessor.java
+++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/AWSSignatureProcessor.java
@@ -43,6 +43,7 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV2;
import org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV4;
import org.apache.hadoop.ozone.s3.header.Credential;
@@ -54,15 +55,15 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Parser to process AWS v4 auth request. Creates string to sign and auth
+ * Parser to process AWS V2 & V4 auth request. Creates string to sign and auth
* header. For more details refer to AWS documentation https://docs.aws
* .amazon.com/general/latest/gr/sigv4-create-canonical-request.html.
**/
@RequestScoped
-public class AWSV4SignatureProcessor implements SignatureProcessor {
+public class AWSSignatureProcessor implements SignatureProcessor {
private final static Logger LOG =
- LoggerFactory.getLogger(AWSV4SignatureProcessor.class);
+ LoggerFactory.getLogger(AWSSignatureProcessor.class);
@Context
private ContainerRequestContext context;
@@ -72,13 +73,12 @@ public class AWSV4SignatureProcessor implements SignatureProcessor {
private String uri;
private String method;
private AuthorizationHeaderV4 v4Header;
+ private AuthorizationHeaderV2 v2Header;
private String stringToSign;
@PostConstruct
public void init()
throws Exception {
- LOG.info("Initializing request header parser");
-
//header map is MUTABLE. It's better to save it here. (with lower case
// keys!!!)
this.headers = new LowerCaseKeyStringMap(new HashMap<>());
@@ -107,10 +107,18 @@ public class AWSV4SignatureProcessor implements SignatureProcessor {
this.uri = context.getUriInfo().getRequestUri().getPath();
this.method = context.getMethod();
- if (v4Header == null) {
- v4Header = new AuthorizationHeaderV4(headers.get(AUTHORIZATION_HEADER));
+ String authHeader = headers.get(AUTHORIZATION_HEADER);
+ String[] split = authHeader.split(" ");
+ if (split[0].equals(AuthorizationHeaderV2.IDENTIFIER)) {
+ if (v2Header == null) {
+ v2Header = new AuthorizationHeaderV2(authHeader);
+ }
+ } else {
+ if (v4Header == null) {
+ v4Header = new AuthorizationHeaderV4(authHeader);
+ }
+ parse();
}
- parse();
}
@@ -320,11 +328,13 @@ public class AWSV4SignatureProcessor implements SignatureProcessor {
}
public String getAwsAccessId() {
- return v4Header.getAccessKeyID();
+ return (v4Header != null ? v4Header.getAccessKeyID() :
+ v2Header != null ? v2Header.getAccessKeyID() : "");
}
public String getSignature() {
- return v4Header.getSignature();
+ return (v4Header != null ? v4Header.getSignature() :
+ v2Header != null ? v2Header.getSignature() : "");
}
public String getStringToSign() throws Exception {
@@ -342,6 +352,11 @@ public class AWSV4SignatureProcessor implements SignatureProcessor {
this.v4Header = v4Header;
}
+ @VisibleForTesting
+ public void setV2Header(AuthorizationHeaderV2 v2Header) {
+ this.v2Header = v2Header;
+ }
+
/**
* A simple map which forces lower case key usage.
*/
diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
index 3cd7b7c..a3042c1 100644
--- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
+++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
@@ -51,7 +51,7 @@ public class OzoneClientProducer {
private OzoneClient client;
@Inject
- private SignatureProcessor v4RequestParser;
+ private SignatureProcessor signatureParser;
@Inject
private OzoneConfiguration ozoneConfiguration;
@@ -76,7 +76,7 @@ public class OzoneClientProducer {
private OzoneClient getClient(OzoneConfiguration config) throws IOException {
try {
- String awsAccessId = v4RequestParser.getAwsAccessId();
+ String awsAccessId = signatureParser.getAwsAccessId();
UserGroupInformation remoteUser =
UserGroupInformation.createRemoteUser(awsAccessId);
if (OzoneSecurityUtil.isSecurityEnabled(config)) {
@@ -85,8 +85,8 @@ public class OzoneClientProducer {
OzoneTokenIdentifier identifier = new OzoneTokenIdentifier();
identifier.setTokenType(S3AUTHINFO);
- identifier.setStrToSign(v4RequestParser.getStringToSign());
- identifier.setSignature(v4RequestParser.getSignature());
+ identifier.setStrToSign(signatureParser.getStringToSign());
+ identifier.setSignature(signatureParser.getSignature());
identifier.setAwsAccessId(awsAccessId);
identifier.setOwner(new Text(awsAccessId));
if (LOG.isTraceEnabled()) {
diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV2.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV2.java
index dfafc3a..fe096ce 100644
--- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV2.java
+++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV2.java
@@ -29,7 +29,7 @@ import static org.apache.commons.lang3.StringUtils.isBlank;
*/
public class AuthorizationHeaderV2 {
- private final static String IDENTIFIER = "AWS";
+ public final static String IDENTIFIER = "AWS";
private String authHeader;
private String identifier;
private String accessKeyID;
diff --git a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV4.java b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV4.java
index 0d9f092..1e48689 100644
--- a/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV4.java
+++ b/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/header/AuthorizationHeaderV4.java
@@ -35,8 +35,8 @@ import static java.time.temporal.ChronoUnit.DAYS;
import static org.apache.commons.lang3.StringUtils.isEmpty;
import static org.apache.commons.lang3.StringUtils.isNotEmpty;
import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.MALFORMED_HEADER;
-import static org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor.AWS4_SIGNING_ALGORITHM;
-import static org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor.DATE_FORMATTER;
+import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.AWS4_SIGNING_ALGORITHM;
+import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.DATE_FORMATTER;
/**
* S3 Authorization header.
diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSV4SignatureProcessor.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSSignatureProcessor.java
similarity index 69%
rename from hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSV4SignatureProcessor.java
rename to hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSSignatureProcessor.java
index 11b3b91..239e285 100644
--- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSV4SignatureProcessor.java
+++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAWSSignatureProcessor.java
@@ -24,6 +24,7 @@ import javax.ws.rs.core.UriInfo;
import java.net.URI;
import org.apache.hadoop.ozone.s3.exception.OS3Exception;
+import org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV2;
import org.apache.hadoop.ozone.s3.header.AuthorizationHeaderV4;
import org.junit.Assert;
@@ -33,10 +34,10 @@ import org.mockito.Mockito;
/**
* Test the Auth parser.
*/
-public class TestAWSV4SignatureProcessor {
+public class TestAWSSignatureProcessor {
@Test
- public void testInitialization() throws Exception {
+ public void testV4Initialization() throws Exception {
MultivaluedMap<String, String> headers = new MultivaluedHashMap<>();
headers.putSingle("Content-Length", "123");
@@ -76,7 +77,7 @@ public class TestAWSV4SignatureProcessor {
Mockito.when(mock.getMethod()).thenReturn("GET");
Mockito.when(mock.getUriInfo()).thenReturn(uriInfo);
- AWSV4SignatureProcessor parser = new AWSV4SignatureProcessor() {
+ AWSSignatureProcessor parser = new AWSSignatureProcessor() {
@Override
void validateSignedHeader(String header, String headerValue)
throws OS3Exception {
@@ -100,4 +101,41 @@ public class TestAWSV4SignatureProcessor {
"f20d4de80af2271545385e8d4c7df608cae70a791c69b97aab1527ed93a0d665",
parser.getStringToSign());
}
+
+ @Test
+ public void testV2Initialization() throws Exception {
+
+ MultivaluedMap<String, String> headers = new MultivaluedHashMap<>();
+ String authHeader = "AWS root:ixWQAgWvJDuqLUqgDG9o4b2HF7c=";
+ headers.putSingle("Authorization", authHeader);
+
+ AuthorizationHeaderV2 parserAuthHeader =
+ new AuthorizationHeaderV2(authHeader);
+
+ MultivaluedMap<String, String> queryParameters = new MultivaluedHashMap<>();
+
+ UriInfo uriInfo = Mockito.mock(UriInfo.class);
+ Mockito.when(uriInfo.getQueryParameters()).thenReturn(queryParameters);
+ Mockito.when(uriInfo.getRequestUri())
+ .thenReturn(new URI("http://localhost/buckets"));
+
+ ContainerRequestContext mock = Mockito.mock(ContainerRequestContext.class);
+ Mockito.when(mock.getHeaders()).thenReturn(headers);
+ Mockito.when(mock.getMethod()).thenReturn("GET");
+ Mockito.when(mock.getUriInfo()).thenReturn(uriInfo);
+
+ AWSSignatureProcessor parser = new AWSSignatureProcessor() {
+ @Override
+ void validateSignedHeader(String header, String headerValue)
+ throws OS3Exception {
+ super.validateSignedHeader(header, headerValue);
+ }
+ };
+ parser.setV2Header(parserAuthHeader);
+ parser.setContext(mock);
+ parser.init();
+
+ Assert.assertEquals("root", parser.getAwsAccessId());
+ Assert.assertEquals("ixWQAgWvJDuqLUqgDG9o4b2HF7c=", parser.getSignature());
+ }
}
\ No newline at end of file
diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java
index 014cb3e..b4a21e3 100644
--- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java
+++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java
@@ -30,7 +30,7 @@ import org.apache.hadoop.ozone.s3.SignatureProcessor;
import org.apache.hadoop.ozone.s3.exception.OS3Exception;
import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
-import static org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor.DATE_FORMATTER;
+import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.DATE_FORMATTER;
import static org.apache.hadoop.ozone.s3.exception.S3ErrorTable.MALFORMED_HEADER;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
diff --git a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/header/TestAuthorizationHeaderV4.java b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/header/TestAuthorizationHeaderV4.java
index e5d5562..5ca1c45 100644
--- a/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/header/TestAuthorizationHeaderV4.java
+++ b/hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/header/TestAuthorizationHeaderV4.java
@@ -26,7 +26,7 @@ import org.junit.Test;
import java.time.LocalDate;
import static java.time.temporal.ChronoUnit.DAYS;
-import static org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor.DATE_FORMATTER;
+import static org.apache.hadoop.ozone.s3.AWSSignatureProcessor.DATE_FORMATTER;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 36/39: HDDS-3986. Frequent failure in
TestCommitWatcher#testReleaseBuffersOnException (#1220)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit d9775d46ada415cd7ff81302b0d84972fd88041c
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Tue Jul 21 20:22:37 2020 +0530
HDDS-3986. Frequent failure in TestCommitWatcher#testReleaseBuffersOnException (#1220)
(cherry picked from commit 402a427f14f5b3d68956e93706de97d1ffae5b94)
---
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
index c3eca6a..007fbdc 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java
@@ -46,6 +46,7 @@ import org.apache.hadoop.ozone.container.ContainerTestHelper;
import org.apache.ratis.protocol.AlreadyClosedException;
import org.apache.ratis.protocol.NotReplicatedException;
import org.apache.ratis.protocol.RaftRetryFailureException;
+import org.apache.ratis.protocol.TimeoutIOException;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -121,8 +122,8 @@ public class TestCommitWatcher {
RatisClientConfig ratisClientConfig =
conf.getObject(RatisClientConfig.class);
- ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
- ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(30));
+ ratisClientConfig.setWriteRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
+ ratisClientConfig.setWatchRequestTimeoutInMs(TimeUnit.SECONDS.toMillis(10));
conf.setFromObject(ratisClientConfig);
conf.set(OzoneConfigKeys.OZONE_CLIENT_CHECKSUM_TYPE, "NONE");
@@ -312,14 +313,17 @@ public class TestCommitWatcher {
watcher.watchForCommit(replies.get(1).getLogIndex() + 100);
Assert.fail("Expected exception not thrown");
} catch(IOException ioe) {
- // with retry count set to lower limit and a lower watch request
+ // with retry count set to noRetry and a lower watch request
// timeout, watch request will eventually
- // fail with RaftRetryFailure exception from ratis client or the client
+ // fail with TimeoutIOException from ratis client or the client
// can itself get AlreadyClosedException from the Ratis Server
+ // and the write may fail with RaftRetryFailureException
Throwable t = HddsClientUtils.checkForException(ioe);
- Assert.assertTrue(t instanceof RaftRetryFailureException ||
- t instanceof AlreadyClosedException ||
- t instanceof NotReplicatedException);
+ Assert.assertTrue("Unexpected exception: " + t.getClass(),
+ t instanceof RaftRetryFailureException ||
+ t instanceof TimeoutIOException ||
+ t instanceof AlreadyClosedException ||
+ t instanceof NotReplicatedException);
}
if (ratisClient.getReplicatedMinCommitIndex() < replies.get(1)
.getLogIndex()) {
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org
[hadoop-ozone] 32/39: HDDS-3987. Encrypted bucket creation failed
with INVALID_REQUEST Encryption cannot be set for bucket links (#1221)
Posted by sa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a commit to branch ozone-0.6.0
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit de093ec2506d828816659273cb31423df56d9cde
Author: Doroszlai, Attila <64...@users.noreply.github.com>
AuthorDate: Mon Jul 20 22:02:55 2020 +0200
HDDS-3987. Encrypted bucket creation failed with INVALID_REQUEST Encryption cannot be set for bucket links (#1221)
(cherry picked from commit 8339b38517c879ac9ce410b8cb0679a462b3d5c7)
---
.../dist/src/main/compose/ozonesecure/test.sh | 4 +++
hadoop-ozone/dist/src/main/compose/testlib.sh | 2 +-
.../dist/src/main/smoketest/ozone-lib/shell.robot | 5 +++
.../smoketest/security/bucket-encryption.robot | 40 ++++++++++++++++++++++
.../om/request/bucket/OMBucketCreateRequest.java | 4 +--
5 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh b/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
index ce50fa0..84de2a9 100755
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
@@ -23,8 +23,12 @@ source "$COMPOSE_DIR/../testlib.sh"
export SECURITY_ENABLED=true
+: ${OZONE_BUCKET_KEY_NAME:=key1}
+
start_docker_env
+execute_command_in_container kms hadoop key create ${OZONE_BUCKET_KEY_NAME}
+
execute_robot_test scm kinit.robot
execute_robot_test scm basic
diff --git a/hadoop-ozone/dist/src/main/compose/testlib.sh b/hadoop-ozone/dist/src/main/compose/testlib.sh
index 56c35c1..5a05633 100755
--- a/hadoop-ozone/dist/src/main/compose/testlib.sh
+++ b/hadoop-ozone/dist/src/main/compose/testlib.sh
@@ -114,7 +114,7 @@ execute_robot_test(){
OUTPUT_PATH="$RESULT_DIR_INSIDE/${OUTPUT_FILE}"
# shellcheck disable=SC2068
docker-compose exec -T "$CONTAINER" mkdir -p "$RESULT_DIR_INSIDE" \
- && docker-compose exec -T "$CONTAINER" robot -v OM_SERVICE_ID:"${OM_SERVICE_ID}" -v SECURITY_ENABLED:"${SECURITY_ENABLED}" -v OM_HA_PARAM:"${OM_HA_PARAM}" ${ARGUMENTS[@]} --log NONE -N "$TEST_NAME" --report NONE "${OZONE_ROBOT_OPTS[@]}" --output "$OUTPUT_PATH" "$SMOKETEST_DIR_INSIDE/$TEST"
+ && docker-compose exec -T "$CONTAINER" robot -v OM_SERVICE_ID:"${OM_SERVICE_ID}" -v SECURITY_ENABLED:"${SECURITY_ENABLED}" -v OM_HA_PARAM:"${OM_HA_PARAM}" -v KEY_NAME:"${OZONE_BUCKET_KEY_NAME}" ${ARGUMENTS[@]} --log NONE -N "$TEST_NAME" --report NONE "${OZONE_ROBOT_OPTS[@]}" --output "$OUTPUT_PATH" "$SMOKETEST_DIR_INSIDE/$TEST"
local -i rc=$?
FULL_CONTAINER_NAME=$(docker-compose ps | grep "_${CONTAINER}_" | head -n 1 | awk '{print $1}')
diff --git a/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
index 2e56ae4..9afc6df 100644
--- a/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
+++ b/hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
@@ -46,3 +46,8 @@ Verify ACL
[arguments] ${object_type} ${object} ${type} ${name} ${acls}
${actual_acls} = Execute ozone sh ${object_type} getacl ${object} | jq -r '.[] | select(.type == "${type}") | select(.name == "${name}") | .aclList[]' | xargs
Should Be Equal ${acls} ${actual_acls}
+
+Create Random Volume
+ ${random} = Generate Random String 5 [LOWER]
+ Execute ozone sh volume create o3://${OM_SERVICE_ID}/vol-${random}
+ [return] vol-${random}
diff --git a/hadoop-ozone/dist/src/main/smoketest/security/bucket-encryption.robot b/hadoop-ozone/dist/src/main/smoketest/security/bucket-encryption.robot
new file mode 100644
index 0000000..e1f96b1
--- /dev/null
+++ b/hadoop-ozone/dist/src/main/smoketest/security/bucket-encryption.robot
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test for bucket encryption
+Library BuiltIn
+Library String
+Resource ../commonlib.robot
+Resource ../lib/os.robot
+Resource ../ozone-lib/shell.robot
+Test Setup Setup Test
+Test Timeout 5 minutes
+
+*** Variables ***
+${KEY_NAME} key1
+${VOLUME}
+
+*** Keywords ***
+Setup Test
+ ${volume} = Create Random Volume
+ Set Suite Variable ${VOLUME} ${volume}
+
+
+*** Test Cases ***
+Create Encrypted Bucket
+ ${output} = Execute ozone sh bucket create -k ${KEY_NAME} o3://${OM_SERVICE_ID}/${VOLUME}/encrypted-bucket
+ Should Not Contain ${output} INVALID_REQUEST
+ Bucket Exists o3://${OM_SERVICE_ID}/${VOLUME}/encrypted-bucket
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
index 71d5458..7c60f61 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java
@@ -115,8 +115,8 @@ public class OMBucketCreateRequest extends OMClientRequest {
newBucketInfo.setBeinfo(getBeinfo(kmsProvider, bucketInfo));
}
- boolean hasSourceVolume = bucketInfo.getSourceVolume() != null;
- boolean hasSourceBucket = bucketInfo.getSourceBucket() != null;
+ boolean hasSourceVolume = bucketInfo.hasSourceVolume();
+ boolean hasSourceBucket = bucketInfo.hasSourceBucket();
if (hasSourceBucket != hasSourceVolume) {
throw new OMException("Both source volume and source bucket are " +
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-commits-help@hadoop.apache.org