You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by xy...@apache.org on 2019/01/18 23:22:40 UTC
[hadoop] branch trunk updated (c26d354 -> 751bc62)
This is an automated email from the ASF dual-hosted git repository.
xyao pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from c26d354 HDDS-983. Rename S3Utils to avoid conflict with HDFS classes. Contributed by Bharat Viswanadham.
add ff61931 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.
add d3920ec HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar.
add 8e6743e HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.
add bfa4929 Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd"
add 914e93b HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by Ajay Kumar.
add e47135d HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar.
add faf53f8 HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar.
add 570b503 Fix merge conflicts
add 0b034b7 HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar.
add 2d26944 HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer.
add 9920506 HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao.
add d451188 HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu Yao.
add 0aab740 HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar.
add 8d7c5f4 HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider. Contributed by Xiaoyu Yao.
add 16e0bb8 HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar.
add e89c35a HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao.
add 61e85d7 HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by Xiaoyu Yao.
add c260c19 HDDS-101. SCM CA: generate CSR for SCM CA clients. Contributed by Xiaoyu Yao.
add 33c274e HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar.
add 8b8a3f5 HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar.
add a28ad7a HDDS-753. SCM security protocol server is not starting. Contributed by Ajay Kumar.
add 53120e2 HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar.
add 6ad794b HDDS-778. Add an interface for CA and Clients for Certificate operations Contributed by Anu Engineer.
add 6d6b1a0 HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar.
add bb4a26c HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar.
add 7e27706 HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao.
add 8253106 HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by Xiaoyu Yao.
add 0c8829a HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. Contributed by Anu Engineer.
add 6d522dc HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar.
add 417951a HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed by Xiaoyu Yao.
add f894d86 HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao.
add 2b11522 HDDS-929. Remove ozone.max.key.len property. Contributed by Ajay Kumar.
add 50c4045 HDDS-805. Block token: Client api changes for block token. Contributed by Ajay Kumar.
add ddaef67 HDDS-937. Create an S3 Auth Table. Contributed by Dinesh Chitlangia.
add 924bea9 HDDS-102. SCM CA: SCM CA server signs certificate for approved CSR. Contributed by Anu Engineer.
add 30bfc9c HDDS-955. SCM CA: Add CA to SCM. Contributed by Anu Engineer.
add 1d5734e HDDS-938. Add Client APIs for using S3 Auth interface. Contributed by Dinesh Chitlangia.
add a5d0fcf HDDS-963. Fix failure in TestOzoneShell due to null check in SecurityConfig. Contributed by Ajay Kumar.
add 8978466 HDDS-945. Fix generics warnings in delegation token. Contributed by Ajay Kumar.
add 0faa570 HDDS-964. Fix test failure in TestOmMetrics. Contributed by Ajay Kumar.
add 0e16cf1 HDDS-970. Fix classnotfound error for bouncy castle classes in OM,SCM init. Contributed by Ajay Kumar.
add 140565f HDDS-967. Fix failures in TestOzoneConfigurationFields. Contributed by Ajay Kumar.
add 01a7f9e HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar.
add c0683ed HDDS-960. Add cli command option for getS3Secret. Contributed by Dinesh Chitlangia.
add 06c83d3 HDDS-984. Fix TestOzoneManagerRatisServer.testIsReadOnlyCapturesAllCmdTypeEnums. Contributed by Xiaoyu Yao.
add 2aaaf12 HDDS-943. Add block token validation in HddsDispatcher/XceiverServer. Contributed by Ajay Kumar.
new 751bc62 Merge branch 'HDDS-4' into trunk
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../security/authentication/util/KerberosUtil.java | 2 +-
.../hdds/scm/ClientCredentialInterceptor.java | 65 +++
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 79 ++-
.../hadoop/hdds/scm/XceiverClientManager.java | 16 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 26 +-
.../hdds/scm/client/ContainerOperationClient.java | 10 +-
hadoop-hdds/common/pom.xml | 18 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 130 +++--
.../java/org/apache/hadoop/hdds/HddsUtils.java | 13 +-
.../org/apache/hadoop/hdds/client/BlockID.java | 8 +-
.../hadoop/hdds/client/ContainerBlockID.java | 10 +-
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 44 ++
.../SCMSecurityProtocolClientSideTranslatorPB.java | 99 ++++
.../hdds/protocolPB/SCMSecurityProtocolPB.java} | 19 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 66 +++
.../hadoop/hdds/protocolPB/package-info.java | 20 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 27 +-
.../apache/hadoop/hdds/scm/XceiverClientSpi.java | 6 +
.../scm/protocol/ScmBlockLocationProtocol.java | 3 +
.../protocol/StorageContainerLocationProtocol.java | 3 +
.../scm/protocolPB/ScmBlockLocationProtocolPB.java | 4 +
.../StorageContainerLocationProtocolPB.java | 4 +
.../hdds/scm/storage/ContainerProtocolCalls.java | 162 ++++--
.../security/exception/SCMSecurityException.java | 78 +++
.../hdds/security/exception/package-info.java | 23 +-
.../hdds/security/token/BlockTokenException.java | 53 ++
.../hdds/security/token/BlockTokenVerifier.java | 131 +++++
.../security/token/OzoneBlockTokenIdentifier.java | 199 ++++++++
.../security/token/OzoneBlockTokenSelector.java | 75 +++
.../hadoop/hdds/security/token/TokenVerifier.java | 38 ++
.../hadoop/hdds/security/token/package-info.java | 22 +
.../hadoop/hdds/security/x509/SecurityConfig.java | 462 +++++++++++++++++
.../x509/certificate/authority/BaseApprover.java | 249 ++++++++++
.../certificate/authority/CertificateApprover.java | 86 ++++
.../certificate/authority/CertificateServer.java | 112 +++++
.../certificate/authority/DefaultApprover.java | 128 +++++
.../certificate/authority/DefaultCAServer.java | 439 ++++++++++++++++
.../authority/PKIProfiles/DefaultCAProfile.java | 46 ++
.../authority/PKIProfiles/DefaultProfile.java | 333 +++++++++++++
.../authority/PKIProfiles/PKIProfile.java | 140 ++++++
.../authority/PKIProfiles/package-info.java | 33 +-
.../x509/certificate/authority/package-info.java | 22 +-
.../x509/certificate/client/CertificateClient.java | 174 +++++++
.../x509/certificate/client/package-info.java | 22 +-
.../x509/certificate/utils/CertificateCodec.java | 280 +++++++++++
.../x509/certificate/utils/package-info.java | 22 +-
.../certificates/utils/CertificateSignRequest.java | 276 ++++++++++
.../certificates/utils/SelfSignedCertificate.java | 238 +++++++++
.../x509/certificates/utils/package-info.java | 22 +-
.../x509/exceptions/CertificateException.java | 53 ++
.../security/x509/exceptions/package-info.java | 23 +-
.../hdds/security/x509/keys/HDDSKeyGenerator.java | 118 +++++
.../hadoop/hdds/security/x509/keys/KeyCodec.java | 366 ++++++++++++++
.../hdds/security/x509/keys/SecurityUtil.java | 138 +++++
.../hdds/security/x509/keys/package-info.java | 23 +-
.../hadoop/hdds/security/x509/package-info.java | 99 ++++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 7 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 16 +
.../OzoneSecurityUtil.java} | 32 +-
.../apache/hadoop/ozone/common/StorageInfo.java | 2 +-
.../main/java/org/apache/ratis/RatisHelper.java | 65 ++-
.../src/main/proto/DatanodeContainerProtocol.proto | 1 +
.../src/main/proto/SCMSecurityProtocol.proto | 66 +++
hadoop-hdds/common/src/main/proto/hdds.proto | 24 +
.../common/src/main/resources/ozone-default.xml | 271 +++++++++-
.../token/TestOzoneBlockTokenIdentifier.java | 313 ++++++++++++
.../hadoop/hdds/security/token/package-info.java | 22 +
.../x509/certificate/authority/MockApprover.java | 57 +++
.../certificate/authority/TestDefaultCAServer.java | 169 +++++++
.../certificate/authority/TestDefaultProfile.java | 364 ++++++++++++++
.../x509/certificate/authority/package-info.java | 22 +-
.../certificate/utils/TestCertificateCodec.java | 218 ++++++++
.../x509/certificate/utils/package-info.java | 23 +-
.../certificates/TestCertificateSignRequest.java | 285 +++++++++++
.../x509/certificates/TestRootCertificate.java | 258 ++++++++++
.../security/x509/certificates/package-info.java | 22 +-
.../security/x509/keys/TestHDDSKeyGenerator.java | 87 ++++
.../hdds/security/x509/keys/TestKeyCodec.java | 231 +++++++++
.../hdds/security/x509/keys/package-info.java | 24 +-
.../hadoop/hdds/security/x509/package-info.java | 24 +-
.../apache/hadoop/utils/db/TestDBStoreBuilder.java | 5 +-
.../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 23 +
.../apache/hadoop/ozone/HddsDatanodeService.java | 30 ++
.../transport/server/GrpcXceiverService.java | 16 +
.../server/ServerCredentialInterceptor.java | 74 +++
.../common/transport/server/XceiverServer.java | 86 ++++
.../common/transport/server/XceiverServerGrpc.java | 44 +-
.../server/ratis/ContainerStateMachine.java | 43 +-
.../transport/server/ratis/XceiverServerRatis.java | 31 +-
.../replication/GrpcReplicationService.java | 1 -
.../protocol/StorageContainerDatanodeProtocol.java | 4 +
.../StorageContainerDatanodeProtocolPB.java | 6 +
.../TestCloseContainerCommandHandler.java | 5 +-
hadoop-hdds/docs/content/S3Commands.md | 41 ++
hadoop-hdds/server-scm/pom.xml | 1 -
.../hdds/scm/pipeline/RatisPipelineUtils.java | 15 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 2 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 123 +++++
.../hdds/scm/server/StorageContainerManager.java | 138 ++++-
.../server/StorageContainerManagerHttpServer.java | 5 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 3 +-
.../scm/server/TestSCMSecurityProtocolServer.java | 60 +++
.../apache/hadoop/ozone/client/ObjectStore.java | 5 +
.../ozone/client/io/BlockOutputStreamEntry.java | 342 +++++++++++++
.../hadoop/ozone/client/io/KeyInputStream.java | 5 +
.../hadoop/ozone/client/io/KeyOutputStream.java | 196 ++------
.../ozone/client/protocol/ClientProtocol.java | 45 +-
.../hadoop/ozone/client/rest/RestClient.java | 52 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 63 ++-
.../hadoop/ozone/client/TestHddsClientUtils.java | 2 +-
hadoop-ozone/common/src/main/bin/start-ozone.sh | 16 +-
hadoop-ozone/common/src/main/bin/stop-ozone.sh | 13 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 111 +++--
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 22 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 7 +
.../apache/hadoop/ozone/om/OzoneManagerLock.java | 24 +-
.../hadoop/ozone/om/helpers/OmKeyLocationInfo.java | 44 +-
.../hadoop/ozone/om/helpers/S3SecretValue.java | 81 +++
.../ozone/om/protocol/OzoneManagerProtocol.java | 20 +-
.../om/protocol/OzoneManagerSecurityProtocol.java | 67 +++
...OzoneManagerProtocolClientSideTranslatorPB.java | 129 +++++
.../om/protocolPB/OzoneManagerProtocolPB.java | 7 +
.../apache/hadoop/ozone/protocolPB/OMPBHelper.java | 38 ++
.../security/OzoneBlockTokenSecretManager.java | 194 ++++++++
.../OzoneDelegationTokenSecretManager.java | 468 +++++++++++++++++
.../security/OzoneDelegationTokenSelector.java | 51 ++
.../hadoop/ozone/security/OzoneSecretKey.java | 176 +++++++
.../hadoop/ozone/security/OzoneSecretManager.java | 279 +++++++++++
.../hadoop/ozone/security/OzoneSecretStore.java | 249 ++++++++++
.../ozone/security/OzoneSecurityException.java} | 53 +-
.../ozone/security/OzoneTokenIdentifier.java | 217 ++++++++
.../package-info.java} | 19 +-
.../src/main/proto/OzoneManagerProtocol.proto | 69 +++
.../security/TestOzoneBlockTokenSecretManager.java | 147 ++++++
.../TestOzoneDelegationTokenSecretManager.java | 217 ++++++++
.../dist/dev-support/bin/dist-layout-stitching | 2 +-
.../main/compose/{ozoneperf => ozonesecure}/.env | 0
.../dist/src/main/compose/ozonesecure/README.md | 22 +
.../main/compose/ozonesecure/docker-compose.yaml | 73 +++
.../src/main/compose/ozonesecure/docker-config | 104 ++++
.../docker-image/docker-krb5/Dockerfile-krb5 | 34 ++
.../ozonesecure/docker-image/docker-krb5/README.md | 34 ++
.../ozonesecure/docker-image/docker-krb5/kadm5.acl | 20 +
.../ozonesecure/docker-image/docker-krb5/krb5.conf | 40 ++
.../docker-image/docker-krb5/launcher.sh | 25 +
.../ozonesecure/docker-image/runner/Dockerfile | 39 ++
.../ozonesecure/docker-image/runner/build.sh | 26 +
.../docker-image/runner/scripts/envtoconf.py | 115 +++++
.../docker-image/runner/scripts/krb5.conf | 38 ++
.../docker-image/runner/scripts/starter.sh | 100 ++++
.../docker-image/runner/scripts/transformation.py | 150 ++++++
.../src/main/smoketest/security/ozone-secure.robot | 111 +++++
hadoop-ozone/dist/src/main/smoketest/test.sh | 2 +
hadoop-ozone/integration-test/pom.xml | 6 +
.../TestContainerStateManagerIntegration.java | 5 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 17 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 26 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 7 +-
.../TestContainerStateMachineIdempotency.java | 9 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 9 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 553 +++++++++++++++++++++
.../hadoop/ozone/TestStorageContainerManager.java | 8 +-
.../ozone/client/CertificateClientTestImpl.java | 148 ++++++
.../apache/hadoop/ozone/client/package-info.java | 24 +-
.../client/rpc/TestContainerStateMachine.java | 151 ++++++
.../ozone/client/rpc/TestOzoneRpcClient.java | 23 +
.../client/rpc/TestOzoneRpcClientAbstract.java | 33 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 238 +++++++++
.../ozone/container/ContainerTestHelper.java | 45 +-
.../container/ozoneimpl/TestOzoneContainer.java | 13 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 190 +++++++
.../ozoneimpl/TestSecureOzoneContainer.java | 227 +++++++++
.../container/server/TestContainerServer.java | 43 +-
...rServer.java => TestSecureContainerServer.java} | 151 +++---
.../hadoop/ozone/ozShell/TestOzoneShell.java | 40 ++
.../hadoop/ozone/scm/TestContainerSmallFile.java | 8 +-
.../scm/TestGetCommittedBlockLengthAndPutKey.java | 8 +-
.../hadoop/ozone/scm/TestXceiverClientManager.java | 19 +-
.../hadoop/ozone/scm/TestXceiverClientMetrics.java | 6 +
.../integration-test/src/test/resources/ssl/ca.crt | 27 +
.../integration-test/src/test/resources/ssl/ca.key | 54 ++
.../src/test/resources/ssl/client.crt | 27 +
.../src/test/resources/ssl/client.csr | 26 +
.../src/test/resources/ssl/client.key | 51 ++
.../src/test/resources/ssl/client.pem | 52 ++
.../src/test/resources/ssl/generate.sh | 34 ++
.../src/test/resources/ssl/server.crt | 27 +
.../src/test/resources/ssl/server.csr | 26 +
.../src/test/resources/ssl/server.key | 51 ++
.../src/test/resources/ssl/server.pem | 52 ++
.../hadoop/ozone/web/OzoneHddsDatanodeService.java | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 1 -
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 75 ++-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 14 +
.../org/apache/hadoop/ozone/om/OzoneManager.java | 525 ++++++++++++++++---
.../hadoop/ozone/om/OzoneManagerHttpServer.java | 5 +-
.../apache/hadoop/ozone/om/S3SecretManager.java} | 26 +-
.../hadoop/ozone/om/S3SecretManagerImpl.java | 82 +++
.../hadoop/ozone/om/exceptions/OMException.java | 1 +
.../protocolPB/OzoneManagerRequestHandler.java | 104 ++++
.../org/apache/hadoop/ozone/web/ozShell/Shell.java | 5 +-
.../ozone/web/ozShell/s3/GetS3SecretHandler.java | 49 ++
.../hadoop/ozone/web/ozShell/s3/S3Commands.java | 60 +++
.../hadoop/ozone/web/ozShell/s3/package-info.java | 21 +
.../hadoop/ozone/om/TestKeyDeletingService.java | 6 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 2 +-
.../ozone/security/TestOzoneManagerBlockToken.java | 251 ++++++++++
.../ozone/security/TestOzoneTokenIdentifier.java | 300 +++++++++++
.../apache/hadoop/ozone/security/package-info.java | 21 +
hadoop-ozone/pom.xml | 3 +
210 files changed, 16057 insertions(+), 1011 deletions(-)
create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/ClientCredentialInterceptor.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java
copy hadoop-hdds/{container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolPB.java => common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolPB.java} (63%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/exception/SCMSecurityException.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/exception/package-info.java (52%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenException.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenIdentifier.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/OzoneBlockTokenSelector.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/TokenVerifier.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/package-info.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/BaseApprover.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateApprover.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateServer.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultApprover.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultCAServer.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultCAProfile.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/DefaultProfile.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/PKIProfile.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/PKIProfiles/package-info.java (52%)
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/CertificateCodec.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/utils/CertificateSignRequest.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/utils/SelfSignedCertificate.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/utils/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/exceptions/CertificateException.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/exceptions/package-info.java (52%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/KeyCodec.java
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/package-info.java (52%)
create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/package-info.java
copy hadoop-hdds/common/src/main/java/org/apache/hadoop/{hdds/scm/protocolPB/ScmBlockLocationProtocolPB.java => ozone/OzoneSecurityUtil.java} (59%)
create mode 100644 hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/token/TestOzoneBlockTokenIdentifier.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/token/package-info.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/MockApprover.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/TestDefaultCAServer.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/TestDefaultProfile.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/authority/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/TestCertificateCodec.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/utils/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/package-info.java (56%)
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestKeyCodec.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/package-info.java (52%)
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/package-info.java (52%)
create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ServerCredentialInterceptor.java
create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServer.java
create mode 100644 hadoop-hdds/docs/content/S3Commands.md
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java
create mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/S3SecretValue.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerSecurityProtocol.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSelector.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretKey.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretManager.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneSecretStore.java
copy hadoop-ozone/{ozone-manager/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java => common/src/main/java/org/apache/hadoop/ozone/security/OzoneSecurityException.java} (66%)
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneTokenIdentifier.java
copy hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/{om/protocolPB/OzoneManagerProtocolPB.java => security/package-info.java} (60%)
create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/TestOzoneBlockTokenSecretManager.java
create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/TestOzoneDelegationTokenSecretManager.java
copy hadoop-ozone/dist/src/main/compose/{ozoneperf => ozonesecure}/.env (100%)
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/README.md
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/README.md
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/krb5.conf
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/launcher.sh
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/Dockerfile
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/build.sh
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/envtoconf.py
create mode 100644 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/krb5.conf
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/transformation.py
create mode 100644 hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/CertificateClientTestImpl.java
copy hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerDatanodeProtocolPB.java => hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/package-info.java (54%)
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachine.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestSecureOzoneRpcClient.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerWithTLS.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
copy hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/server/{TestContainerServer.java => TestSecureContainerServer.java} (70%)
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/ca.crt
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/ca.key
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/client.crt
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/client.csr
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/client.key
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/client.pem
create mode 100755 hadoop-ozone/integration-test/src/test/resources/ssl/generate.sh
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/server.crt
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/server.csr
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/server.key
create mode 100644 hadoop-ozone/integration-test/src/test/resources/ssl/server.pem
copy hadoop-ozone/{common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolPB.java => ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3SecretManager.java} (52%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3SecretManagerImpl.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/s3/GetS3SecretHandler.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/s3/S3Commands.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/s3/package-info.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/TestOzoneManagerBlockToken.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/TestOzoneTokenIdentifier.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/package-info.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/01: Merge branch 'HDDS-4' into trunk
Posted by xy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xyao pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 751bc62df7bb8ffba524189634dec8e93a90ab32
Merge: 2aaaf12 c26d354
Author: Xiaoyu Yao <xy...@apache.org>
AuthorDate: Fri Jan 18 14:40:21 2019 -0800
Merge branch 'HDDS-4' into trunk
BUILDING.txt | 3 -
.../apache/hadoop/io/erasurecode/jni_xor_decoder.c | 2 +
.../hadoop/hdds/scm/XceiverClientManager.java | 11 +-
.../hdds/scm/client/ContainerOperationClient.java | 10 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 2 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 17 +-
.../container/common/impl/ContainerDataYaml.java | 17 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 4 +-
.../apache/hadoop/hdfs/DFSOpsCountStatistics.java | 14 ++
.../apache/hadoop/hdfs/DistributedFileSystem.java | 28 ++++
.../hadoop/hdfs/TestDistributedFileSystem.java | 121 ++++++++++++++
.../ozone/client/io/BlockOutputStreamEntry.java | 57 +++----
.../hadoop/ozone/client/io/KeyInputStream.java | 2 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 60 +++----
.../TestContainerStateMachineIdempotency.java | 2 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 2 +-
.../hadoop/ozone/scm/TestContainerSmallFile.java | 8 +-
.../scm/TestGetCommittedBlockLengthAndPutKey.java | 6 +-
.../hadoop/ozone/scm/TestXceiverClientManager.java | 52 ++++++-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 20 ++-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 4 +-
.../apache/hadoop/ozone/s3/util/ContinueToken.java | 173 +++++++++++++++++++++
.../{S3utils.java => RangeHeaderParserUtil.java} | 74 +--------
.../hadoop/ozone/s3/endpoint/TestBucketGet.java | 49 +++++-
.../hadoop/ozone/s3/util/TestContinueToken.java | 50 ++++++
...S3utils.java => TestRangeHeaderParserUtil.java} | 20 +--
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 7 +
.../component/instance/ComponentInstance.java | 8 +-
.../src/main/resources/yarn-default.xml | 4 +-
.../rmapp/attempt/RMAppAttemptImpl.java | 14 +-
.../rmapp/attempt/TestRMAppAttemptTransitions.java | 80 +++++++++-
.../storage/DataGeneratorForTest.java | 2 +-
...reator.java => HBaseTimelineSchemaCreator.java} | 10 +-
.../timelineservice/storage/SchemaCreator.java | 28 ++++
.../storage/TimelineSchemaCreator.java | 80 ++++++++++
.../storage/DummyTimelineSchemaCreator.java | 29 ++++
.../storage/TestTimelineSchemaCreator.java | 41 +++++
.../hadoop-yarn-ui/src/main/webapp/bower.json | 2 +-
38 files changed, 889 insertions(+), 224 deletions(-)
diff --cc hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java
index 6dcdad9,0000000..9c07cf6
mode 100644,000000..100644
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntry.java
@@@ -1,359 -1,0 +1,342 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.client.io;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.XceiverClientManager;
+import org.apache.hadoop.hdds.scm.XceiverClientSpi;
++import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.hdds.scm.storage.BlockOutputStream;
+import org.apache.hadoop.hdds.security.token.OzoneBlockTokenIdentifier;
+import org.apache.hadoop.ozone.common.Checksum;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Helper class used inside {@link BlockOutputStream}.
+ * */
+public class BlockOutputStreamEntry extends OutputStream {
+
+ private OutputStream outputStream;
+ private BlockID blockID;
+ private final String key;
+ private final XceiverClientManager xceiverClientManager;
- private final XceiverClientSpi xceiverClient;
++ private final Pipeline pipeline;
+ private final Checksum checksum;
+ private final String requestId;
+ private final int chunkSize;
+ // total number of bytes that should be written to this stream
+ private final long length;
+ // the current position of this stream 0 <= currentPosition < length
+ private long currentPosition;
+ private Token<OzoneBlockTokenIdentifier> token;
+
+ private final long streamBufferFlushSize;
+ private final long streamBufferMaxSize;
+ private final long watchTimeout;
+ private List<ByteBuffer> bufferList;
+
+ private BlockOutputStreamEntry(BlockID blockID, String key,
+ XceiverClientManager xceiverClientManager,
- XceiverClientSpi xceiverClient, String requestId, int chunkSize,
++ Pipeline pipeline, String requestId, int chunkSize,
+ long length, long streamBufferFlushSize, long streamBufferMaxSize,
+ long watchTimeout, List<ByteBuffer> bufferList, Checksum checksum,
+ Token<OzoneBlockTokenIdentifier> token) {
+ this.outputStream = null;
+ this.blockID = blockID;
+ this.key = key;
+ this.xceiverClientManager = xceiverClientManager;
- this.xceiverClient = xceiverClient;
++ this.pipeline = pipeline;
+ this.requestId = requestId;
+ this.chunkSize = chunkSize;
+ this.token = token;
+ this.length = length;
+ this.currentPosition = 0;
+ this.streamBufferFlushSize = streamBufferFlushSize;
+ this.streamBufferMaxSize = streamBufferMaxSize;
+ this.watchTimeout = watchTimeout;
+ this.bufferList = bufferList;
+ this.checksum = checksum;
+ }
+
- /**
- * For testing purpose, taking a some random created stream instance.
- *
- * @param outputStream a existing writable output stream
- * @param length the length of data to write to the stream
- */
- BlockOutputStreamEntry(OutputStream outputStream, long length,
- Checksum checksum) {
- this.outputStream = outputStream;
- this.blockID = null;
- this.key = null;
- this.xceiverClientManager = null;
- this.xceiverClient = null;
- this.requestId = null;
- this.chunkSize = -1;
- this.token = null;
- this.length = length;
- this.currentPosition = 0;
- streamBufferFlushSize = 0;
- streamBufferMaxSize = 0;
- bufferList = null;
- watchTimeout = 0;
- this.checksum = checksum;
- }
-
+ long getLength() {
+ return length;
+ }
+
+ Token<OzoneBlockTokenIdentifier> getToken() {
+ return token;
+ }
+
+ long getRemaining() {
+ return length - currentPosition;
+ }
+
++ /**
++ * BlockOutputStream is initialized in this function. This makes sure that
++ * xceiverClient initialization is not done during preallocation and only
++ * done when data is written.
++ * @throws IOException if xceiverClient initialization fails
++ */
+ private void checkStream() throws IOException {
+ if (this.outputStream == null) {
+ if (getToken() != null) {
+ UserGroupInformation.getCurrentUser().addToken(getToken());
+ }
+ this.outputStream =
+ new BlockOutputStream(blockID, key, xceiverClientManager,
- xceiverClient, requestId, chunkSize, streamBufferFlushSize,
++ pipeline, requestId, chunkSize, streamBufferFlushSize,
+ streamBufferMaxSize, watchTimeout, bufferList, checksum);
+ }
+ }
+
++
+ @Override
+ public void write(int b) throws IOException {
+ checkStream();
+ outputStream.write(b);
+ this.currentPosition += 1;
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ checkStream();
+ outputStream.write(b, off, len);
+ this.currentPosition += len;
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (this.outputStream != null) {
+ this.outputStream.flush();
+ }
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (this.outputStream != null) {
+ this.outputStream.close();
+ // after closing the chunkOutPutStream, blockId would have been
+ // reconstructed with updated bcsId
+ if (this.outputStream instanceof BlockOutputStream) {
+ this.blockID = ((BlockOutputStream) outputStream).getBlockID();
+ }
+ }
+ }
+
+ long getTotalSuccessfulFlushedData() throws IOException {
+ if (this.outputStream instanceof BlockOutputStream) {
+ BlockOutputStream out = (BlockOutputStream) this.outputStream;
+ blockID = out.getBlockID();
+ return out.getTotalSuccessfulFlushedData();
+ } else if (outputStream == null) {
+ // For a pre allocated block for which no write has been initiated,
+ // the OutputStream will be null here.
+ // In such cases, the default blockCommitSequenceId will be 0
+ return 0;
+ }
+ throw new IOException("Invalid Output Stream for Key: " + key);
+ }
+
+ long getWrittenDataLength() throws IOException {
+ if (this.outputStream instanceof BlockOutputStream) {
+ BlockOutputStream out = (BlockOutputStream) this.outputStream;
+ return out.getWrittenDataLength();
+ } else if (outputStream == null) {
+ // For a pre allocated block for which no write has been initiated,
+ // the OutputStream will be null here.
+ // In such cases, the default blockCommitSequenceId will be 0
+ return 0;
+ }
+ throw new IOException("Invalid Output Stream for Key: " + key);
+ }
+
- void cleanup() throws IOException{
++ void cleanup(boolean invalidateClient) throws IOException {
+ checkStream();
+ if (this.outputStream instanceof BlockOutputStream) {
+ BlockOutputStream out = (BlockOutputStream) this.outputStream;
- out.cleanup();
++ out.cleanup(invalidateClient);
+ }
+ }
+
+ void writeOnRetry(long len) throws IOException {
+ checkStream();
+ if (this.outputStream instanceof BlockOutputStream) {
+ BlockOutputStream out = (BlockOutputStream) this.outputStream;
+ out.writeOnRetry(len);
+ this.currentPosition += len;
+ } else {
+ throw new IOException("Invalid Output Stream for Key: " + key);
+ }
+ }
+
+ /**
+ * Builder class for ChunkGroupOutputStreamEntry.
+ * */
+ public static class Builder {
+
+ private BlockID blockID;
+ private String key;
+ private XceiverClientManager xceiverClientManager;
- private XceiverClientSpi xceiverClient;
++ private Pipeline pipeline;
+ private String requestId;
+ private int chunkSize;
+ private long length;
+ private long streamBufferFlushSize;
+ private long streamBufferMaxSize;
+ private long watchTimeout;
+ private List<ByteBuffer> bufferList;
+ private Token<OzoneBlockTokenIdentifier> token;
+ private Checksum checksum;
+
+ public Builder setChecksum(Checksum cs) {
+ this.checksum = cs;
+ return this;
+ }
+
+ public Builder setBlockID(BlockID bID) {
+ this.blockID = bID;
+ return this;
+ }
+
+ public Builder setKey(String keys) {
+ this.key = keys;
+ return this;
+ }
+
+ public Builder setXceiverClientManager(XceiverClientManager
+ xClientManager) {
+ this.xceiverClientManager = xClientManager;
+ return this;
+ }
+
- public Builder setXceiverClient(XceiverClientSpi client) {
- this.xceiverClient = client;
++ public Builder setPipeline(Pipeline pipeline) {
++ this.pipeline = pipeline;
+ return this;
+ }
+
+ public Builder setRequestId(String request) {
+ this.requestId = request;
+ return this;
+ }
+
+ public Builder setChunkSize(int cSize) {
+ this.chunkSize = cSize;
+ return this;
+ }
+
+ public Builder setLength(long len) {
+ this.length = len;
+ return this;
+ }
+
+ public Builder setStreamBufferFlushSize(long bufferFlushSize) {
+ this.streamBufferFlushSize = bufferFlushSize;
+ return this;
+ }
+
+ public Builder setStreamBufferMaxSize(long bufferMaxSize) {
+ this.streamBufferMaxSize = bufferMaxSize;
+ return this;
+ }
+
+ public Builder setWatchTimeout(long timeout) {
+ this.watchTimeout = timeout;
+ return this;
+ }
+
+ public Builder setBufferList(List<ByteBuffer> bufferList) {
+ this.bufferList = bufferList;
+ return this;
+ }
+
+ public Builder setToken(Token<OzoneBlockTokenIdentifier> bToken) {
+ this.token = bToken;
+ return this;
+ }
+
+ public BlockOutputStreamEntry build() {
+ return new BlockOutputStreamEntry(blockID, key,
- xceiverClientManager, xceiverClient, requestId, chunkSize,
++ xceiverClientManager, pipeline, requestId, chunkSize,
+ length, streamBufferFlushSize, streamBufferMaxSize, watchTimeout,
+ bufferList, checksum, token);
+ }
+ }
+
+ public OutputStream getOutputStream() {
+ return outputStream;
+ }
+
+ public BlockID getBlockID() {
+ return blockID;
+ }
+
+ public String getKey() {
+ return key;
+ }
+
+ public XceiverClientManager getXceiverClientManager() {
+ return xceiverClientManager;
+ }
+
- public XceiverClientSpi getXceiverClient() {
- return xceiverClient;
++ public Pipeline getPipeline() {
++ return pipeline;
+ }
+
+ public Checksum getChecksum() {
+ return checksum;
+ }
+
+ public String getRequestId() {
+ return requestId;
+ }
+
+ public int getChunkSize() {
+ return chunkSize;
+ }
+
+ public long getCurrentPosition() {
+ return currentPosition;
+ }
+
+ public long getStreamBufferFlushSize() {
+ return streamBufferFlushSize;
+ }
+
+ public long getStreamBufferMaxSize() {
+ return streamBufferMaxSize;
+ }
+
+ public long getWatchTimeout() {
+ return watchTimeout;
+ }
+
+ public List<ByteBuffer> getBufferList() {
+ return bufferList;
+ }
+
+ public void setCurrentPosition(long curPosition) {
+ this.currentPosition = curPosition;
+ }
+}
+
+
diff --cc hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java
index 1874017,cbbfed8..22efab3
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java
@@@ -21,10 -21,11 +21,9 @@@ import com.google.common.annotations.Vi
import com.google.common.base.Preconditions;
import org.apache.hadoop.fs.FSExceptionMessages;
import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result;
--import org.apache.hadoop.hdds.client.BlockID;
++import org.apache.hadoop.hdds.scm.XceiverClientSpi;
import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException;
import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline;
-import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
--import org.apache.hadoop.hdds.scm.storage.BlockOutputStream;
import org.apache.hadoop.ozone.common.Checksum;
import org.apache.hadoop.ozone.om.helpers.*;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType;
@@@ -36,7 -36,7 +34,8 @@@ import org.apache.hadoop.hdds.scm.conta
.StorageContainerException;
import org.apache.hadoop.hdds.scm.protocolPB
.StorageContainerLocationProtocolClientSideTranslatorPB;
+import org.apache.hadoop.security.UserGroupInformation;
+ import org.apache.ratis.protocol.AlreadyClosedException;
import org.apache.ratis.protocol.RaftRetryFailureException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@@ -215,27 -201,14 +201,27 @@@ public class KeyOutputStream extends Ou
throws IOException {
ContainerWithPipeline containerWithPipeline = scmClient
.getContainerWithPipeline(subKeyInfo.getContainerID());
- streamEntries.add(new BlockOutputStreamEntry(subKeyInfo.getBlockID(),
- keyArgs.getKeyName(), xceiverClientManager,
- containerWithPipeline.getPipeline(), requestID, chunkSize,
- subKeyInfo.getLength(), streamBufferFlushSize, streamBufferMaxSize,
- watchTimeout, bufferList, checksum));
+ UserGroupInformation.getCurrentUser().addToken(subKeyInfo.getToken());
+ XceiverClientSpi xceiverClient =
+ xceiverClientManager.acquireClient(containerWithPipeline.getPipeline());
+ BlockOutputStreamEntry.Builder builder =
+ new BlockOutputStreamEntry.Builder()
+ .setBlockID(subKeyInfo.getBlockID())
+ .setKey(keyArgs.getKeyName())
+ .setXceiverClientManager(xceiverClientManager)
- .setXceiverClient(xceiverClient)
++ .setPipeline(containerWithPipeline.getPipeline())
+ .setRequestId(requestID)
+ .setChunkSize(chunkSize)
+ .setLength(subKeyInfo.getLength())
+ .setStreamBufferFlushSize(streamBufferFlushSize)
+ .setStreamBufferMaxSize(streamBufferMaxSize)
+ .setWatchTimeout(watchTimeout)
+ .setBufferList(bufferList)
+ .setChecksum(checksum)
+ .setToken(subKeyInfo.getToken());
+ streamEntries.add(builder.build());
}
-
@Override
public void write(int b) throws IOException {
byte[] buf = new byte[1];
@@@ -383,10 -360,10 +373,10 @@@
long totalSuccessfulFlushedData =
streamEntry.getTotalSuccessfulFlushedData();
//set the correct length for the current stream
- streamEntry.currentPosition = totalSuccessfulFlushedData;
+ streamEntry.setCurrentPosition(totalSuccessfulFlushedData);
long bufferedDataLen = computeBufferData();
// just clean up the current stream.
- streamEntry.cleanup();
+ streamEntry.cleanup(retryFailure);
if (bufferedDataLen > 0) {
// If the data is still cached in the underlying stream, we need to
// allocate new block and write this data in the datanode.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org