You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by zh...@apache.org on 2019/07/02 03:18:53 UTC
[hadoop] branch HDFS-13572 updated (edd741d -> f324188)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a change to branch HDFS-13572
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard edd741d HDFS-13643. Implement basic async rpc client
add 041e7a7 HDFS-14598. Findbugs warning caused by HDFS-12487. Contributed by He Xiaoqiao.
add 366f3de HADOOP-16390. escape javadoc in S3AUtils public methods
add 48d7f00 HDFS-14590. [SBN Read] Add the document link to the top page. Contributed by Takanobu Asanuma.
add ab0b180 HDFS-12345 Add Dynamometer to hadoop-tools, a tool for scale testing the HDFS NameNode with real metadata and workloads. Contributed by Erik Krogen.
add 049bb42 HDDS-1723. Create new OzoneManagerLock class. (#1006)
add 4b50981 HDDS-1709. TestScmSafeNode is flaky. Contributed by Elek, Marton.
add a79bdf7 HDDS-1727. Use generation of resourceName for locks in OzoneManagerLock. (#1014)
add 062eb60 YARN-9477. Implement VE discovery using libudev. Contributed by Peter Bacsko.
add 0a37ccf HDDS-1691 : RDBTable#isExist should use Rocksdb#keyMayExist (#1013)
add 1ac967a YARN-6055. ContainersMonitorImpl need be adjusted when NM resource changed. Contributed by Inigo Goiri.
add 4848280 HDDS-1638. Implement Key Write Requests to use Cache and DoubleBuffer. (#956)
add be80334 YARN-9639. DecommissioningNodesWatcher cause memory leak. Contributed by Bilwa S T.
add 041e0c0 HDFS-14036. RBF: Add hdfs-rbf-default.xml to HdfsConfiguration by default. Contributed by Takanobu Asanuma.
add 4a21224 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao.
add cbae241 YARN-9623. Auto adjust max queue length of app activities to make sure activities on all nodes can be covered. Contributed by Tao Yang.
add 5e7caf1 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023
add f09c31a Revert "YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023"
add 9fd3c70 HDDS-1639. Restructure documentation pages for better understanding
add f02b0e1 YARN-9581. Add support for get multiple RM webapp URLs. Contributed by Prabhu Joseph
add 49c5e8a HDDS-1672. Improve locking in OzoneManager. (#1016)
add 29465bf YARN-9560. Restructure DockerLinuxContainerRuntime to extend OCIContainerRuntime. Contributed by Eric Badger, Jim Brennan, Craig Condit
add da56899 HDDS-1733. Fix Ozone documentation (#1031)
add d203045 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward.
add 1e727cf HDDS-1730. Implement File CreateDirectory Request to use Cache and Do… (#1026)
add 9df6275 HDDS-1555. Disable install snapshot for ContainerStateMachine. Contributed by Siddharth Wagle. (#846)
add 1f75660 HDDS-373. Genconf tool must generate ozone-site.xml with sample values (#1025)
add b1dafc3 YARN-9661:Fix typo in LocalityMulticastAMRMProxyPolicy.java and AbstractConfigurableFederationPolicy.java (#1042)
add f8d62a9 HDDS-1258. Fix error propagation for SCM protocol
add db674a0 HDDS-1685. Recon: Add support for "start" query param to containers and containers/{id} endpoints.
add d8bac50 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015)
add 2f4b37b HDDS-1667. Docker compose file may referring to incorrect docker image name. Contributed by Eric Yang.
add 570eee3 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi.
new f324188 HDFS-13643. Implement basic async rpc client
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (edd741d)
\
N -- N -- N refs/heads/HDFS-13572 (f324188)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev-support/bin/dist-layout-stitching | 1 +
.../assemblies/hadoop-dynamometer-blockgen.xml | 35 +
.../assemblies/hadoop-dynamometer-infra.xml | 35 +
.../assemblies/hadoop-dynamometer-workload.xml | 35 +
.../resources/assemblies/hadoop-dynamometer.xml | 73 ++
.../src/main/resources/assemblies/hadoop-tools.xml | 24 +-
hadoop-dist/pom.xml | 1 +
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 72 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 7 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 6 +-
.../hadoop/hdds/scm/exceptions/package-info.java | 4 +-
...lockLocationProtocolClientSideTranslatorPB.java | 94 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 6 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 6 +
.../org/apache/hadoop/ozone/lock/LockManager.java | 5 +-
...lockLocationProtocolServerSideTranslatorPB.java | 155 ++-
.../java/org/apache/hadoop/utils/db/RDBTable.java | 6 +-
.../java/org/apache/hadoop/utils/db/Table.java | 8 +
.../org/apache/hadoop/utils/db/TypedTable.java | 5 +
.../src/main/proto/ScmBlockLocationProtocol.proto | 36 +-
.../common/src/main/resources/ozone-default.xml | 8 +
.../exceptions/TestSCMExceptionResultCodes.java | 52 +
.../org/apache/hadoop/ozone/audit/DummyAction.java | 1 -
.../org/apache/hadoop/utils/db/TestRDBStore.java | 36 +
.../server/ratis/ContainerStateMachine.java | 28 +-
.../transport/server/ratis/XceiverServerRatis.java | 51 +-
hadoop-hdds/docs/config.yaml | 26 +-
hadoop-hdds/docs/content/Concepts.md | 108 --
hadoop-hdds/docs/content/Hdds.md | 65 --
hadoop-hdds/docs/content/OzoneManager.md | 77 --
.../docs/content/OzoneSecurityArchitecture.md | 110 --
hadoop-hdds/docs/content/RealCluster.md | 74 --
hadoop-hdds/docs/content/SetupSecureOzone.md | 98 --
hadoop-hdds/docs/content/_index.md | 28 +-
hadoop-hdds/docs/content/beyond/Containers.md | 218 ++++
.../{Dozone.md => beyond/DockerCheatSheet.md} | 46 +-
.../docs/content/{ => beyond}/RunningWithHDFS.md | 17 +-
.../content/{CommandShell.md => beyond/Tools.md} | 62 +-
.../docs/content/{genconf.md => beyond/_index.md} | 20 +-
.../docs/content/concept/ContainerMetadata.png | Bin 0 -> 98493 bytes
hadoop-hdds/docs/content/concept/Datanodes.md | 75 ++
.../docs/content/concept/FunctionalOzone.png | Bin 0 -> 55334 bytes
hadoop-hdds/docs/content/concept/Hdds.md | 52 +
hadoop-hdds/docs/content/concept/Overview.md | 81 ++
hadoop-hdds/docs/content/concept/OzoneBlock.png | Bin 0 -> 4650 bytes
hadoop-hdds/docs/content/concept/OzoneManager.md | 87 ++
.../docs/content/{genconf.md => concept/_index.md} | 21 +-
.../docs/content/concept/ozoneBlockDiagram.png | Bin 0 -> 45218 bytes
.../docs/content/{ => interface}/JavaApi.md | 12 +-
.../docs/content/{ => interface}/OzoneFS.md | 30 +-
hadoop-hdds/docs/content/{ => interface}/S3.md | 9 +-
.../content/{genconf.md => interface/_index.md} | 13 +-
.../docs/content/{ => recipe}/Prometheus.md | 11 +-
.../docs/content/{ => recipe}/SparkOzoneFSK8S.md | 12 +-
.../docs/content/{genconf.md => recipe/_index.md} | 16 +-
.../recipe}/prometheus-key-allocate.png | Bin
.../docs/{static => content/recipe}/prometheus.png | Bin
.../docs/content/security/SecuityWithRanger.md | 43 +
hadoop-hdds/docs/content/security/SecureOzone.md | 177 +++
.../docs/content/security/SecuringDatanodes.md | 70 ++
hadoop-hdds/docs/content/security/SecuringS3.md | 61 ++
hadoop-hdds/docs/content/security/SecuringTDE.md | 66 ++
hadoop-hdds/docs/content/security/SecurityAcls.md | 85 ++
hadoop-hdds/docs/content/security/_index.md | 36 +
.../docs/content/{ => shell}/BucketCommands.md | 6 +-
hadoop-hdds/docs/content/shell/Format.md | 69 ++
.../docs/content/{ => shell}/KeyCommands.md | 11 +-
.../docs/content/{ => shell}/VolumeCommands.md | 6 +-
.../docs/content/{genconf.md => shell/_index.md} | 12 +-
.../{BuildingSources.md => start/FromSource.md} | 27 +-
hadoop-hdds/docs/content/start/Kubernetes.md | 52 +
hadoop-hdds/docs/content/start/Minikube.md | 69 ++
.../docs/content/{Settings.md => start/OnPrem.md} | 73 +-
.../docs/content/{ => start}/RunningViaDocker.md | 29 +-
.../docs/content/start/StartFromDockerHub.md | 111 ++
hadoop-hdds/docs/content/start/_index.md | 88 ++
hadoop-hdds/docs/content/start/docker.png | Bin 0 -> 7002 bytes
hadoop-hdds/docs/content/start/hadoop.png | Bin 0 -> 4923 bytes
hadoop-hdds/docs/content/start/k8s.png | Bin 0 -> 6270 bytes
hadoop-hdds/docs/content/start/minikube.png | Bin 0 -> 5764 bytes
.../docs/content/{ => tools}/AuditParser.md | 4 +-
hadoop-hdds/docs/content/{ => tools}/Freon.md | 4 +-
.../docs/content/{genconf.md => tools/Genconf.md} | 4 +-
hadoop-hdds/docs/content/{ => tools}/SCMCLI.md | 4 +-
hadoop-hdds/docs/content/tools/TestTools.md | 228 ++++
.../docs/content/{genconf.md => tools/Tools.md} | 13 +-
.../{ozone-logo.png => ozone-logo-small.png} | Bin
hadoop-hdds/docs/static/ozone-usage.png | Bin 0 -> 108376 bytes
.../themes/ozonedoc/layouts/_default/section.html | 60 ++
.../themes/ozonedoc/layouts/_default/single.html | 47 +-
.../docs/themes/ozonedoc/layouts/index.html | 2 +-
.../themes/ozonedoc/layouts/partials/header.html | 4 +-
.../themes/ozonedoc/layouts/partials/sidebar.html | 13 +-
.../ozonedoc/layouts/shortcodes/buttonlink.html} | 16 +-
.../{_default/single.html => shortcodes/card.html} | 36 +-
.../ozonedoc/layouts/shortcodes/jumbotron.html} | 19 +-
.../ozonedoc/layouts/shortcodes/requirements.html} | 18 +-
.../docs/themes/ozonedoc/static/css/ozonedoc.css | 17 +
hadoop-hdds/pom.xml | 2 +-
.../org/apache/hadoop/hdfs/HdfsConfiguration.java | 1 +
.../federation/router/RouterClientProtocol.java | 2 +-
.../fs/contract/router/RouterHDFSContract.java | 1 -
.../contract/router/web/RouterWebHDFSContract.java | 1 -
.../server/federation/MiniRouterDFSCluster.java | 11 +-
.../server/blockmanagement/DatanodeDescriptor.java | 7 +-
.../PendingReconstructionBlocks.java | 4 +-
.../hadoop/hdfs/server/datanode/DiskBalancer.java | 31 +-
.../hdfs/server/diskbalancer/TestDiskBalancer.java | 11 +-
.../org/apache/hadoop/ozone/audit/OMAction.java | 1 -
.../apache/hadoop/ozone/om/OMMetadataManager.java | 12 +
.../apache/hadoop/ozone/om/OzoneManagerLock.java | 269 -----
.../hadoop/ozone/om/S3SecretManagerImpl.java | 9 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 17 +-
.../hadoop/ozone/om/helpers/OmKeyLocationInfo.java | 25 +
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 378 +++++++
.../hadoop/ozone/om/lock/OzoneManagerLockUtil.java | 75 ++
.../apache/hadoop/ozone/om/lock}/package-info.java | 7 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 13 +-
.../src/main/proto/OzoneManagerProtocol.proto | 7 +
.../hadoop/ozone/om/lock/TestOzoneManagerLock.java | 348 ++++++
.../apache/hadoop/ozone/om/lock}/package-info.java | 12 +-
hadoop-ozone/dist/pom.xml | 4 -
.../src/test/compose/docker-compose.yaml | 8 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 2 +-
.../apache/hadoop/ozone/om/TestOzoneManager.java | 63 ++
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 4 +-
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 18 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 47 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 112 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 71 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 98 +-
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 21 +-
.../apache/hadoop/ozone/om/S3BucketManager.java | 6 +
.../hadoop/ozone/om/S3BucketManagerImpl.java | 21 +-
.../java/org/apache/hadoop/ozone/om/ScmClient.java | 6 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 88 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 7 +
.../ozone/om/ratis/OzoneManagerRatisClient.java | 59 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 8 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 114 --
.../om/ratis/utils/OzoneManagerRatisUtils.java | 18 +
.../hadoop/ozone/om/request/OMClientRequest.java | 2 +-
.../hadoop/ozone/om/request/RequestAuditor.java | 37 +-
.../om/request/bucket/OMBucketCreateRequest.java | 17 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 8 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 8 +-
.../om/request/file/OMDirectoryCreateRequest.java | 282 +++++
.../ozone/om/request/file}/package-info.java | 14 +-
.../om/request/key/OMAllocateBlockRequest.java | 248 +++++
.../ozone/om/request/key/OMKeyCommitRequest.java | 192 ++++
.../ozone/om/request/key/OMKeyCreateRequest.java | 378 +++++++
.../ozone/om/request/key/OMKeyDeleteRequest.java | 166 +++
.../ozone/om/request/key/OMKeyRenameRequest.java | 205 ++++
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 205 ++++
.../hadoop/ozone/om/request/key}/package-info.java | 14 +-
.../om/request/volume/OMVolumeCreateRequest.java | 18 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 44 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 29 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 6 +-
.../response/file/OMDirectoryCreateResponse.java | 67 ++
.../ozone/om/response/file}/package-info.java | 14 +-
.../om/response/key/OMAllocateBlockResponse.java | 59 +
.../ozone/om/response/key/OMKeyCommitResponse.java | 66 ++
.../ozone/om/response/key/OMKeyCreateResponse.java | 61 ++
.../ozone/om/response/key/OMKeyDeleteResponse.java | 78 ++
.../ozone/om/response/key/OMKeyRenameResponse.java | 67 ++
.../ozone/om/response/key}/package-info.java | 14 +-
.../OzoneManagerHARequestHandlerImpl.java | 6 +
...OzoneManagerProtocolServerSideTranslatorPB.java | 60 +-
.../hadoop/ozone/om/TestOzoneManagerLock.java | 193 ----
.../om/ratis/TestOzoneManagerStateMachine.java | 292 -----
.../ozone/om/request/TestOMRequestUtils.java | 69 ++
.../request/bucket/TestOMBucketCreateRequest.java | 1 +
.../request/file/TestOMDirectoryCreateRequest.java | 327 ++++++
.../ozone/om/request/file}/package-info.java | 14 +-
.../om/request/key/TestOMAllocateBlockRequest.java | 241 +++++
.../om/request/key/TestOMKeyCommitRequest.java | 300 ++++++
.../om/request/key/TestOMKeyCreateRequest.java | 325 ++++++
.../om/request/key/TestOMKeyDeleteRequest.java | 166 +++
.../om/request/key/TestOMKeyRenameRequest.java | 225 ++++
.../ozone/om/request/key/TestOMKeyRequest.java | 151 +++
.../hadoop/ozone/om/request/key}/package-info.java | 14 +-
.../file/TestOMDirectoryCreateResponse.java | 123 +++
.../ozone/om/response/file}/package-info.java | 14 +-
.../response/key/TestOMAllocateBlockResponse.java | 93 ++
.../om/response/key/TestOMKeyCommitResponse.java | 108 ++
.../om/response/key/TestOMKeyCreateResponse.java | 92 ++
.../om/response/key/TestOMKeyDeleteResponse.java | 163 +++
.../om/response/key/TestOMKeyRenameResponse.java | 148 +++
.../ozone/om/response/key/TestOMKeyResponse.java | 75 ++
.../ozone/om/response/key}/package-info.java | 14 +-
.../apache/hadoop/ozone/recon/ReconConstants.java | 4 +
.../ozone/recon/api/ContainerKeyService.java | 34 +-
.../recon/spi/ContainerDBServiceProvider.java | 23 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 90 +-
.../ozone/recon/api/TestContainerKeyService.java | 119 +-
.../impl/TestContainerDBServiceProviderImpl.java | 108 ++
hadoop-ozone/pom.xml | 9 +-
.../GenerateOzoneRequiredConfigurations.java | 16 +-
.../TestGenerateOzoneRequiredConfigurations.java | 16 +
hadoop-project/pom.xml | 8 +-
hadoop-project/src/site/site.xml | 1 +
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 4 +-
.../hadoop-dynamometer-blockgen/pom.xml | 131 +++
.../src/main/bash/generate-block-lists.sh | 38 +
.../dynamometer/blockgenerator/BlockInfo.java | 136 +++
.../blockgenerator/GenerateBlockImagesDriver.java | 140 +++
.../GenerateDNBlockInfosReducer.java | 99 ++
.../dynamometer/blockgenerator/XMLParser.java | 154 +++
.../blockgenerator/XMLParserMapper.java | 81 ++
.../dynamometer/blockgenerator}/package-info.java | 13 +-
.../dynamometer/blockgenerator/TestBlockGen.java | 89 ++
.../dynamometer/blockgenerator/TestXMLParser.java | 70 ++
.../test/resources/fsimage_0000000000000061740.xml | 553 ++++++++++
.../hadoop-dynamometer-dist/pom.xml | 128 +++
.../hadoop-dynamometer-infra/pom.xml | 201 ++++
.../src/main/bash/create-slim-hadoop-tar.sh | 53 +
.../src/main/bash/parse-metrics.sh | 84 ++
.../src/main/bash/start-dynamometer-cluster.sh | 41 +
.../src/main/bash/upload-fsimage.sh | 84 ++
.../apache/hadoop/tools/dynamometer/AMOptions.java | 316 ++++++
.../dynamometer/AllowAllImpersonationProvider.java | 24 +-
.../tools/dynamometer/ApplicationMaster.java | 885 +++++++++++++++
.../BlockPlacementPolicyAlwaysSatisfied.java | 53 +
.../apache/hadoop/tools/dynamometer/Client.java | 1132 ++++++++++++++++++++
.../hadoop/tools/dynamometer/DynoConstants.java | 130 +++
.../hadoop/tools/dynamometer/DynoInfraUtils.java | 584 ++++++++++
.../hadoop/tools/dynamometer/DynoResource.java | 72 ++
.../tools/dynamometer/SimulatedDataNodes.java | 182 ++++
.../hadoop/tools/dynamometer}/package-info.java | 16 +-
.../src/main/resources/start-component.sh | 344 ++++++
.../tools/dynamometer/TestDynamometerInfra.java | 542 ++++++++++
.../tools/dynamometer/TestDynoInfraUtils.java | 66 ++
.../src/test/resources/blocks/dn0-a-0-r-00000 | 34 +
.../src/test/resources/blocks/dn1-a-0-r-00001 | 33 +
.../src/test/resources/blocks/dn2-a-0-r-00002 | 33 +
.../test/resources/conf/etc/hadoop/core-site.xml | 20 +
.../test/resources/conf/etc/hadoop/hdfs-site.xml | 20 +
.../resources/conf/etc/hadoop/log4j.properties | 40 +-
.../src/test/resources/hadoop_3_1/VERSION | 7 +
.../hadoop_3_1/fsimage_0000000000000061740 | Bin 0 -> 13795 bytes
.../hadoop_3_1/fsimage_0000000000000061740.md5 | 1 +
.../src/test/resources/yarn-site.xml | 19 +
.../hadoop-dynamometer-workload/pom.xml | 146 +++
.../src/main/bash/parse-start-timestamp.sh | 52 +
.../src/main/bash/start-workload.sh | 42 +
.../workloadgenerator/CreateFileMapper.java | 146 +++
.../workloadgenerator/VirtualInputFormat.java | 57 +
.../workloadgenerator/VirtualInputSplit.java | 37 +-
.../workloadgenerator/VirtualRecordReader.java | 78 ++
.../workloadgenerator/WorkloadDriver.java | 208 ++++
.../workloadgenerator/WorkloadMapper.java | 59 +
.../audit/AuditCommandParser.java | 57 +
.../audit/AuditLogDirectParser.java | 145 +++
.../audit/AuditLogHiveTableParser.java | 70 ++
.../audit/AuditReplayCommand.java | 163 +++
.../workloadgenerator/audit/AuditReplayMapper.java | 290 +++++
.../workloadgenerator/audit/AuditReplayThread.java | 310 ++++++
.../audit/NoSplitTextInputFormat.java | 47 +
.../workloadgenerator/audit}/package-info.java | 14 +-
.../workloadgenerator}/package-info.java | 15 +-
.../workloadgenerator/TestWorkloadGenerator.java | 136 +++
.../audit/TestAuditLogDirectParser.java | 143 +++
.../src/test/resources/audit_trace_direct/audit0 | 6 +
.../src/test/resources/audit_trace_hive/audit0 | 6 +
hadoop-tools/{ => hadoop-dynamometer}/pom.xml | 33 +-
hadoop-tools/pom.xml | 1 +
.../hadoop-yarn/dev-support/findbugs-exclude.xml | 6 +
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
.../hadoop/yarn/webapp/util/WebAppUtils.java | 12 +-
.../src/main/resources/yarn-default.xml | 2 +-
.../hadoop-yarn/hadoop-yarn-csi/pom.xml | 2 -
.../AbstractConfigurableFederationPolicy.java | 2 +-
.../LocalityMulticastAMRMProxyPolicy.java | 6 +-
.../hadoop-yarn-server-nodemanager/pom.xml | 4 +
.../server/nodemanager/LinuxContainerExecutor.java | 5 +-
.../server/nodemanager/NodeStatusUpdaterImpl.java | 5 +
.../yarn/server/nodemanager/ResourceView.java | 8 +
.../amrmproxy/FederationInterceptor.java | 5 +
.../launcher/ContainerCleanup.java | 4 +-
.../resources/gpu/GpuResourceHandlerImpl.java | 4 +-
.../linux/runtime/DockerLinuxContainerRuntime.java | 351 +-----
.../linux/runtime/OCIContainerRuntime.java | 374 +++++++
.../monitor/ContainersMonitor.java | 6 +
.../monitor/ContainersMonitorImpl.java | 49 +-
.../resourceplugin/com/nec/NECVEPlugin.java | 56 +-
.../resourceplugin/com/nec/UdevUtil.java | 100 ++
.../resourceplugin/com/nec/VEDeviceDiscoverer.java | 143 +++
.../deviceframework/DeviceResourceHandlerImpl.java | 4 +-
.../server/nodemanager/TestNodeStatusUpdater.java | 103 ++
.../amrmproxy/TestFederationInterceptor.java | 1 +
.../linux/runtime/TestDockerContainerRuntime.java | 41 +-
.../resourceplugin/com/nec/TestNECVEPlugin.java | 58 +-
.../com/nec/TestVEDeviceDiscoverer.java | 283 +++++
.../DecommissioningNodesWatcher.java | 5 +
.../resourcemanager/ResourceTrackerService.java | 1 +
.../scheduler/activities/ActivitiesManager.java | 52 +-
.../scheduler/capacity/CapacityScheduler.java | 8 +
.../activities/TestActivitiesManager.java | 63 ++
299 files changed, 19508 insertions(+), 2868 deletions(-)
create mode 100644 hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-blockgen.xml
create mode 100644 hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-infra.xml
create mode 100644 hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer-workload.xml
create mode 100644 hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/exceptions/TestSCMExceptionResultCodes.java
delete mode 100644 hadoop-hdds/docs/content/Concepts.md
delete mode 100644 hadoop-hdds/docs/content/Hdds.md
delete mode 100644 hadoop-hdds/docs/content/OzoneManager.md
delete mode 100644 hadoop-hdds/docs/content/OzoneSecurityArchitecture.md
delete mode 100644 hadoop-hdds/docs/content/RealCluster.md
delete mode 100644 hadoop-hdds/docs/content/SetupSecureOzone.md
create mode 100644 hadoop-hdds/docs/content/beyond/Containers.md
rename hadoop-hdds/docs/content/{Dozone.md => beyond/DockerCheatSheet.md} (60%)
rename hadoop-hdds/docs/content/{ => beyond}/RunningWithHDFS.md (83%)
rename hadoop-hdds/docs/content/{CommandShell.md => beyond/Tools.md} (55%)
copy hadoop-hdds/docs/content/{genconf.md => beyond/_index.md} (69%)
create mode 100644 hadoop-hdds/docs/content/concept/ContainerMetadata.png
create mode 100644 hadoop-hdds/docs/content/concept/Datanodes.md
create mode 100644 hadoop-hdds/docs/content/concept/FunctionalOzone.png
create mode 100644 hadoop-hdds/docs/content/concept/Hdds.md
create mode 100644 hadoop-hdds/docs/content/concept/Overview.md
create mode 100644 hadoop-hdds/docs/content/concept/OzoneBlock.png
create mode 100644 hadoop-hdds/docs/content/concept/OzoneManager.md
copy hadoop-hdds/docs/content/{genconf.md => concept/_index.md} (67%)
create mode 100644 hadoop-hdds/docs/content/concept/ozoneBlockDiagram.png
rename hadoop-hdds/docs/content/{ => interface}/JavaApi.md (95%)
rename hadoop-hdds/docs/content/{ => interface}/OzoneFS.md (90%)
rename hadoop-hdds/docs/content/{ => interface}/S3.md (95%)
copy hadoop-hdds/docs/content/{genconf.md => interface/_index.md} (77%)
rename hadoop-hdds/docs/content/{ => recipe}/Prometheus.md (91%)
rename hadoop-hdds/docs/content/{ => recipe}/SparkOzoneFSK8S.md (98%)
copy hadoop-hdds/docs/content/{genconf.md => recipe/_index.md} (76%)
rename hadoop-hdds/docs/{static => content/recipe}/prometheus-key-allocate.png (100%)
rename hadoop-hdds/docs/{static => content/recipe}/prometheus.png (100%)
create mode 100644 hadoop-hdds/docs/content/security/SecuityWithRanger.md
create mode 100644 hadoop-hdds/docs/content/security/SecureOzone.md
create mode 100644 hadoop-hdds/docs/content/security/SecuringDatanodes.md
create mode 100644 hadoop-hdds/docs/content/security/SecuringS3.md
create mode 100644 hadoop-hdds/docs/content/security/SecuringTDE.md
create mode 100644 hadoop-hdds/docs/content/security/SecurityAcls.md
create mode 100644 hadoop-hdds/docs/content/security/_index.md
rename hadoop-hdds/docs/content/{ => shell}/BucketCommands.md (98%)
create mode 100644 hadoop-hdds/docs/content/shell/Format.md
rename hadoop-hdds/docs/content/{ => shell}/KeyCommands.md (98%)
rename hadoop-hdds/docs/content/{ => shell}/VolumeCommands.md (98%)
copy hadoop-hdds/docs/content/{genconf.md => shell/_index.md} (77%)
rename hadoop-hdds/docs/content/{BuildingSources.md => start/FromSource.md} (81%)
create mode 100644 hadoop-hdds/docs/content/start/Kubernetes.md
create mode 100644 hadoop-hdds/docs/content/start/Minikube.md
rename hadoop-hdds/docs/content/{Settings.md => start/OnPrem.md} (73%)
rename hadoop-hdds/docs/content/{ => start}/RunningViaDocker.md (76%)
create mode 100644 hadoop-hdds/docs/content/start/StartFromDockerHub.md
create mode 100644 hadoop-hdds/docs/content/start/_index.md
create mode 100644 hadoop-hdds/docs/content/start/docker.png
create mode 100644 hadoop-hdds/docs/content/start/hadoop.png
create mode 100644 hadoop-hdds/docs/content/start/k8s.png
create mode 100644 hadoop-hdds/docs/content/start/minikube.png
rename hadoop-hdds/docs/content/{ => tools}/AuditParser.md (98%)
rename hadoop-hdds/docs/content/{ => tools}/Freon.md (98%)
copy hadoop-hdds/docs/content/{genconf.md => tools/Genconf.md} (96%)
rename hadoop-hdds/docs/content/{ => tools}/SCMCLI.md (97%)
create mode 100644 hadoop-hdds/docs/content/tools/TestTools.md
copy hadoop-hdds/docs/content/{genconf.md => tools/Tools.md} (75%)
rename hadoop-hdds/docs/static/{ozone-logo.png => ozone-logo-small.png} (100%)
create mode 100644 hadoop-hdds/docs/static/ozone-usage.png
create mode 100644 hadoop-hdds/docs/themes/ozonedoc/layouts/_default/section.html
copy hadoop-hdds/docs/{content/genconf.md => themes/ozonedoc/layouts/shortcodes/buttonlink.html} (75%)
copy hadoop-hdds/docs/themes/ozonedoc/layouts/{_default/single.html => shortcodes/card.html} (55%)
copy hadoop-hdds/docs/{content/genconf.md => themes/ozonedoc/layouts/shortcodes/jumbotron.html} (75%)
rename hadoop-hdds/docs/{content/genconf.md => themes/ozonedoc/layouts/shortcodes/requirements.html} (75%)
delete mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerLock.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLockUtil.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock}/package-info.java (89%)
create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/lock/TestOzoneManagerLock.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/lock}/package-info.java (83%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file/OMDirectoryCreateResponse.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/file}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyDeleteResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyRenameResponse.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key}/package-info.java (75%)
delete mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerLock.java
delete mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequest.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMAllocateBlockRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyDeleteRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRenameRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRequest.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/file/TestOMDirectoryCreateResponse.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/file}/package-info.java (75%)
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMAllocateBlockResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMKeyCommitResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMKeyCreateResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMKeyDeleteResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMKeyRenameResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key/TestOMKeyResponse.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/key}/package-info.java (75%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/pom.xml
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/bash/generate-block-lists.sh
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator/BlockInfo.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator/GenerateBlockImagesDriver.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator/GenerateDNBlockInfosReducer.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator/XMLParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator/XMLParserMapper.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/main/java/org/apache/hadoop/tools/dynamometer/blockgenerator}/package-info.java (72%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/test/java/org/apache/hadoop/tools/dynamometer/blockgenerator/TestBlockGen.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/test/java/org/apache/hadoop/tools/dynamometer/blockgenerator/TestXMLParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen/src/test/resources/fsimage_0000000000000061740.xml
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-dist/pom.xml
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/pom.xml
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/bash/create-slim-hadoop-tar.sh
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/bash/parse-metrics.sh
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/bash/start-dynamometer-cluster.sh
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/bash/upload-fsimage.sh
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/AMOptions.java
copy hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/package-info.java => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/AllowAllImpersonationProvider.java (57%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/ApplicationMaster.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/Client.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/DynoConstants.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/DynoInfraUtils.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/DynoResource.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer}/package-info.java (61%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/resources/start-component.sh
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/java/org/apache/hadoop/tools/dynamometer/TestDynamometerInfra.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/java/org/apache/hadoop/tools/dynamometer/TestDynoInfraUtils.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/blocks/dn0-a-0-r-00000
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/blocks/dn1-a-0-r-00001
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/blocks/dn2-a-0-r-00002
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/conf/etc/hadoop/core-site.xml
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/conf/etc/hadoop/hdfs-site.xml
copy hadoop-hdds/docs/config.yaml => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/conf/etc/hadoop/log4j.properties (57%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/hadoop_3_1/VERSION
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/hadoop_3_1/fsimage_0000000000000061740
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/hadoop_3_1/fsimage_0000000000000061740.md5
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/test/resources/yarn-site.xml
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/pom.xml
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/bash/parse-start-timestamp.sh
create mode 100755 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/bash/start-workload.sh
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/CreateFileMapper.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/VirtualInputFormat.java
copy hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/package-info.java => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/VirtualInputSplit.java (51%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/VirtualRecordReader.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/WorkloadDriver.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/WorkloadMapper.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditCommandParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditLogDirectParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditLogHiveTableParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditReplayCommand.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditReplayMapper.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/AuditReplayThread.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/NoSplitTextInputFormat.java
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit}/package-info.java (67%)
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions => hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/main/java/org/apache/hadoop/tools/dynamometer/workloadgenerator}/package-info.java (66%)
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/test/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/TestWorkloadGenerator.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/test/java/org/apache/hadoop/tools/dynamometer/workloadgenerator/audit/TestAuditLogDirectParser.java
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/test/resources/audit_trace_direct/audit0
create mode 100644 hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload/src/test/resources/audit_trace_hive/audit0
copy hadoop-tools/{ => hadoop-dynamometer}/pom.xml (60%)
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/UdevUtil.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/VEDeviceDiscoverer.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/com/nec/TestVEDeviceDiscoverer.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/01: HDFS-13643. Implement basic async rpc client
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HDFS-13572
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit f324188551dfc714c29f1ae35b82f82ab592d4e9
Author: zhangduo <zh...@apache.org>
AuthorDate: Mon Jun 4 21:54:45 2018 +0800
HDFS-13643. Implement basic async rpc client
---
.../hadoop-client-minicluster/pom.xml | 4 +
hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 29 +++-
.../hdfs/ipc/BufferCallBeforeInitHandler.java | 100 ++++++++++++
.../main/java/org/apache/hadoop/hdfs/ipc/Call.java | 132 ++++++++++++++++
.../org/apache/hadoop/hdfs/ipc/ConnectionId.java | 71 +++++++++
.../apache/hadoop/hdfs/ipc/HdfsRpcController.java | 74 +++++++++
.../java/org/apache/hadoop/hdfs/ipc/IPCUtil.java | 34 ++++
.../java/org/apache/hadoop/hdfs/ipc/RpcClient.java | 128 +++++++++++++++
.../org/apache/hadoop/hdfs/ipc/RpcConnection.java | 153 ++++++++++++++++++
.../apache/hadoop/hdfs/ipc/RpcDuplexHandler.java | 175 +++++++++++++++++++++
.../org/apache/hadoop/hdfs/ipc/TestAsyncIPC.java | 88 +++++++++++
.../apache/hadoop/hdfs/ipc/TestRpcProtocolPB.java | 27 ++++
.../org/apache/hadoop/hdfs/ipc/TestServer.java | 58 +++++++
.../src/test/proto/test_rpc.proto | 35 +++++
hadoop-project/pom.xml | 2 +-
.../hadoop-yarn/hadoop-yarn-csi/pom.xml | 2 -
16 files changed, 1104 insertions(+), 8 deletions(-)
diff --git a/hadoop-client-modules/hadoop-client-minicluster/pom.xml b/hadoop-client-modules/hadoop-client-minicluster/pom.xml
index 918b374..594e28c 100644
--- a/hadoop-client-modules/hadoop-client-minicluster/pom.xml
+++ b/hadoop-client-modules/hadoop-client-minicluster/pom.xml
@@ -115,6 +115,10 @@
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</exclusion>
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
index 8769bef..863f700 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
@@ -39,6 +39,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<artifactId>okhttp</artifactId>
</dependency>
<dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<scope>provided</scope>
@@ -64,11 +69,6 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<scope>test</scope>
</dependency>
<dependency>
- <groupId>io.netty</groupId>
- <artifactId>netty-all</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-netty</artifactId>
<scope>test</scope>
@@ -163,6 +163,25 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
</source>
</configuration>
</execution>
+ <execution>
+ <id>compile-test-protoc</id>
+ <goals>
+ <goal>test-protoc</goal>
+ </goals>
+ <configuration>
+ <protocVersion>${protobuf.version}</protocVersion>
+ <protocCommand>${protoc.path}</protocCommand>
+ <imports>
+ <param>${basedir}/src/test/proto</param>
+ </imports>
+ <source>
+ <directory>${basedir}/src/test/proto</directory>
+ <includes>
+ <include>test_rpc.proto</include>
+ </includes>
+ </source>
+ </configuration>
+ </execution>
</executions>
</plugin>
<plugin>
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/BufferCallBeforeInitHandler.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/BufferCallBeforeInitHandler.java
new file mode 100644
index 0000000..89433e9
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/BufferCallBeforeInitHandler.java
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import io.netty.channel.ChannelDuplexHandler;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.hadoop.classification.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class BufferCallBeforeInitHandler extends ChannelDuplexHandler {
+
+ private enum BufferCallAction {
+ FLUSH, FAIL
+ }
+
+ public static final class BufferCallEvent {
+
+ public final BufferCallAction action;
+
+ public final IOException error;
+
+ private BufferCallEvent(BufferCallBeforeInitHandler.BufferCallAction action,
+ IOException error) {
+ this.action = action;
+ this.error = error;
+ }
+
+ public static BufferCallBeforeInitHandler.BufferCallEvent success() {
+ return SUCCESS_EVENT;
+ }
+
+ public static BufferCallBeforeInitHandler.BufferCallEvent fail(
+ IOException error) {
+ return new BufferCallEvent(BufferCallAction.FAIL, error);
+ }
+ }
+
+ private static final BufferCallEvent SUCCESS_EVENT =
+ new BufferCallEvent(BufferCallAction.FLUSH, null);
+
+ private final Map<Integer, Call> id2Call = new HashMap<>();
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg,
+ ChannelPromise promise) {
+ if (msg instanceof Call) {
+ Call call = (Call) msg;
+ id2Call.put(call.getId(), call);
+ // The call is already in track so here we set the write operation as
+ // success.
+ // We will fail the call directly if we can not write it out.
+ promise.trySuccess();
+ } else {
+ ctx.write(msg, promise);
+ }
+ }
+
+ @Override
+ public void userEventTriggered(ChannelHandlerContext ctx, Object evt)
+ throws Exception {
+ if (evt instanceof BufferCallEvent) {
+ BufferCallEvent bcEvt = (BufferCallBeforeInitHandler.BufferCallEvent) evt;
+ switch (bcEvt.action) {
+ case FLUSH:
+ for (Call call : id2Call.values()) {
+ ctx.write(call);
+ }
+ break;
+ case FAIL:
+ for (Call call : id2Call.values()) {
+ call.setException(bcEvt.error);
+ }
+ break;
+ }
+ ctx.flush();
+ ctx.pipeline().remove(this);
+ } else {
+ ctx.fireUserEventTriggered(evt);
+ }
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/Call.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/Call.java
new file mode 100644
index 0000000..14a35af
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/Call.java
@@ -0,0 +1,132 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import com.google.protobuf.Message;
+import com.google.protobuf.RpcCallback;
+import java.io.IOException;
+import org.apache.hadoop.classification.InterfaceAudience;
+
+@InterfaceAudience.Private
+class Call {
+ private final int id;
+
+ private final String protocolName;
+
+ private final long protocolVersion;
+
+ private final String methodName;
+
+ private final Message param;
+
+ private final Message responseDefaultType;
+
+ private volatile Message response;
+
+ private volatile IOException error;
+
+ private boolean done;
+
+ private final RpcCallback<Call> callback;
+
+ Call(int id, String protocolName, long protocolVersion, String methodName,
+ Message param, Message responseDefaultType, RpcCallback<Call> callback) {
+ this.id = id;
+ this.protocolName = protocolName;
+ this.protocolVersion = protocolVersion;
+ this.methodName = methodName;
+ this.param = param;
+ this.responseDefaultType = responseDefaultType;
+ this.callback = callback;
+ }
+
+ private void callComplete() {
+ callback.run(this);
+ }
+
+ /**
+ * Set the exception when there is an error. Notify the caller the call is
+ * done.
+ *
+ * @param error exception thrown by the call; either local or remote
+ */
+ void setException(IOException error) {
+ synchronized (this) {
+ if (done) {
+ return;
+ }
+ this.done = true;
+ this.error = error;
+ }
+ callComplete();
+ }
+
+ /**
+ * Set the return value when there is no error. Notify the caller the call is
+ * done.
+ *
+ * @param response return value of the call.
+ * @param cells Can be null
+ */
+ void setResponse(Message response) {
+ synchronized (this) {
+ if (done) {
+ return;
+ }
+ this.done = true;
+ this.response = response;
+ }
+ callComplete();
+ }
+
+ int getId() {
+ return id;
+ }
+
+ String getProtocolName() {
+ return protocolName;
+ }
+
+ long getProtocolVersion() {
+ return protocolVersion;
+ }
+
+ String getMethodName() {
+ return methodName;
+ }
+
+ Message getParam() {
+ return param;
+ }
+
+ Message getResponseDefaultType() {
+ return responseDefaultType;
+ }
+
+ Message getResponse() {
+ return response;
+ }
+
+ IOException getError() {
+ return error;
+ }
+
+ boolean isDone() {
+ return done;
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/ConnectionId.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/ConnectionId.java
new file mode 100644
index 0000000..111b925
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/ConnectionId.java
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import java.net.InetSocketAddress;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.security.UserGroupInformation;
+
+@InterfaceAudience.Private
+class ConnectionId {
+
+ private static final int PRIME = 16777619;
+
+ private final UserGroupInformation ticket;
+ private final String protocolName;
+ private final InetSocketAddress address;
+
+ public ConnectionId(UserGroupInformation ticket, String protocolName,
+ InetSocketAddress address) {
+ this.ticket = ticket;
+ this.protocolName = protocolName;
+ this.address = address;
+ }
+
+ UserGroupInformation getTicket() {
+ return ticket;
+ }
+
+ String getProtocolName() {
+ return protocolName;
+ }
+
+ InetSocketAddress getAddress() {
+ return address;
+ }
+
+ @Override
+ public int hashCode() {
+ int h = ticket == null ? 0 : ticket.hashCode();
+ h = PRIME * h + protocolName.hashCode();
+ h = PRIME * h + address.hashCode();
+ return h;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj instanceof ConnectionId) {
+ ConnectionId id = (ConnectionId) obj;
+ return address.equals(id.address) &&
+ ((ticket != null && ticket.equals(id.ticket)) ||
+ (ticket == id.ticket)) &&
+ protocolName.equals(id.protocolName);
+ }
+ return false;
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/HdfsRpcController.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/HdfsRpcController.java
new file mode 100644
index 0000000..71ac3ef
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/HdfsRpcController.java
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+
+import java.io.IOException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+
+@InterfaceAudience.Private
+public class HdfsRpcController implements RpcController {
+
+ private IOException error;
+
+ @Override
+ public void reset() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public boolean failed() {
+ return error != null;
+ }
+
+ @Override
+ public String errorText() {
+ return error != null ? error.getMessage() : null;
+ }
+
+ @Override
+ public void startCancel() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void setFailed(String reason) {
+ this.error = new IOException(reason);
+ }
+
+ public void setException(IOException error) {
+ this.error = error;
+ }
+
+ public IOException getException() {
+ return error;
+ }
+
+ @Override
+ public boolean isCanceled() {
+ return false;
+ }
+
+ @Override
+ public void notifyOnCancel(RpcCallback<Object> callback) {
+ throw new UnsupportedOperationException();
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/IPCUtil.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/IPCUtil.java
new file mode 100644
index 0000000..db46bdb
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/IPCUtil.java
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import java.io.IOException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+
+@InterfaceAudience.Private
+class IPCUtil {
+
+ static IOException toIOE(Throwable t) {
+ if (t instanceof IOException) {
+ return (IOException) t;
+ } else {
+ return new IOException(t);
+ }
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcClient.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcClient.java
new file mode 100644
index 0000000..4792173
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcClient.java
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Message;
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcChannel;
+
+import io.netty.channel.Channel;
+import io.netty.channel.EventLoopGroup;
+import io.netty.channel.nio.NioEventLoopGroup;
+import io.netty.channel.socket.nio.NioSocketChannel;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.ipc.ClientId;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.hadoop.ipc.Server.AuthProtocol;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * The protobuf based rpc client.
+ */
+@InterfaceAudience.Private
+public class RpcClient implements Closeable {
+
+ private final byte[] clientId;
+
+ private final EventLoopGroup group = new NioEventLoopGroup();
+
+ private final Class<? extends Channel> channelClass = NioSocketChannel.class;
+
+ private final AtomicInteger callIdCnt = new AtomicInteger(0);
+
+ private final ConcurrentMap<ConnectionId, RpcConnection> connections =
+ new ConcurrentHashMap<>();
+
+ public RpcClient() {
+ this.clientId = ClientId.getClientId();
+ }
+
+ private int nextCallId() {
+ int id, next;
+ do {
+ id = callIdCnt.get();
+ next = id < Integer.MAX_VALUE ? id + 1 : 0;
+ } while (!callIdCnt.compareAndSet(id, next));
+ return id;
+ }
+
+ private void onCallFinished(Call call, HdfsRpcController hrc,
+ InetSocketAddress addr, RpcCallback<Message> callback) {
+ IOException error = call.getError();
+ if (error != null) {
+ if (error instanceof RemoteException) {
+ error.fillInStackTrace();
+ }
+ hrc.setException(error);
+ callback.run(null);
+ } else {
+ callback.run(call.getResponse());
+ }
+ }
+
+ private void callMethod(String protocolName, long protocolVersion,
+ Descriptors.MethodDescriptor md, HdfsRpcController hrc, Message param,
+ Message returnType, UserGroupInformation ugi, InetSocketAddress addr,
+ RpcCallback<Message> callback) {
+ Call call =
+ new Call(nextCallId(), protocolName, protocolVersion, md.getName(),
+ param, returnType, c -> onCallFinished(c, hrc, addr, callback));
+ ConnectionId remoteId = new ConnectionId(ugi, protocolName, addr);
+ connections
+ .computeIfAbsent(remoteId,
+ k -> new RpcConnection(this, k, AuthProtocol.NONE))
+ .sendRequest(call);
+ }
+
+ public RpcChannel createRpcChannel(Class<?> protocol, InetSocketAddress addr,
+ UserGroupInformation ugi) {
+ String protocolName = RPC.getProtocolName(protocol);
+ long protocolVersion = RPC.getProtocolVersion(protocol);
+ return (method, controller, request, responsePrototype, done) -> callMethod(
+ protocolName, protocolVersion, method, (HdfsRpcController) controller,
+ request, responsePrototype, ugi, addr, done);
+ }
+
+ byte[] getClientId() {
+ return clientId;
+ }
+
+ EventLoopGroup getGroup() {
+ return group;
+ }
+
+ Class<? extends Channel> getChannelClass() {
+ return channelClass;
+ }
+
+ @Override
+ public void close() throws IOException {
+ connections.values().forEach(c -> c.shutdown());
+ connections.clear();
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcConnection.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcConnection.java
new file mode 100644
index 0000000..5e7b482
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcConnection.java
@@ -0,0 +1,153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import static org.apache.hadoop.ipc.RpcConstants.CONNECTION_CONTEXT_CALL_ID;
+
+import com.google.protobuf.CodedOutputStream;
+import io.netty.bootstrap.Bootstrap;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufOutputStream;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelOption;
+import io.netty.channel.ChannelPipeline;
+import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
+import java.io.IOException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.ipc.BufferCallBeforeInitHandler.BufferCallEvent;
+import org.apache.hadoop.ipc.RPC.RpcKind;
+import org.apache.hadoop.ipc.RpcConstants;
+import org.apache.hadoop.ipc.Server.AuthProtocol;
+import org.apache.hadoop.ipc.protobuf.IpcConnectionContextProtos.IpcConnectionContextProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto.OperationProto;
+import org.apache.hadoop.security.SaslRpcServer.AuthMethod;
+import org.apache.hadoop.util.ProtoUtil;
+
+/**
+ * The connection to remote server.
+ */
+@InterfaceAudience.Private
+class RpcConnection {
+
+ final RpcClient rpcClient;
+
+ final ConnectionId remoteId;
+
+ private final AuthProtocol authProtocol;
+
+ private Channel channel;
+
+ public RpcConnection(RpcClient rpcClient, ConnectionId remoteId,
+ AuthProtocol authProtocol) {
+ this.rpcClient = rpcClient;
+ this.remoteId = remoteId;
+ this.authProtocol = authProtocol;
+ }
+
+ private void writeConnectionHeader(Channel ch) {
+ ByteBuf header = ch.alloc().buffer(7);
+ header.writeBytes(RpcConstants.HEADER.duplicate());
+ header.writeByte(RpcConstants.CURRENT_VERSION);
+ header.writeByte(0); // service class
+ header.writeByte(authProtocol.callId);
+ ch.writeAndFlush(header);
+ }
+
+ private void writeConnectionContext(Channel ch) throws IOException {
+ RpcRequestHeaderProto connectionContextHeader =
+ ProtoUtil.makeRpcRequestHeader(RpcKind.RPC_PROTOCOL_BUFFER,
+ OperationProto.RPC_FINAL_PACKET, CONNECTION_CONTEXT_CALL_ID,
+ RpcConstants.INVALID_RETRY_COUNT, rpcClient.getClientId());
+ int headerSize = connectionContextHeader.getSerializedSize();
+ IpcConnectionContextProto message = ProtoUtil.makeIpcConnectionContext(
+ remoteId.getProtocolName(), remoteId.getTicket(), AuthMethod.SIMPLE);
+ int messageSize = message.getSerializedSize();
+
+ int totalSize =
+ CodedOutputStream.computeRawVarint32Size(headerSize) + headerSize +
+ CodedOutputStream.computeRawVarint32Size(messageSize) + messageSize;
+ ByteBuf buf = ch.alloc().buffer(totalSize + 4);
+ buf.writeInt(totalSize);
+ ByteBufOutputStream out = new ByteBufOutputStream(buf);
+ connectionContextHeader.writeDelimitedTo(out);
+ message.writeDelimitedTo(out);
+ ch.writeAndFlush(buf);
+ }
+
+ private void established(Channel ch) throws IOException {
+ ChannelPipeline p = ch.pipeline();
+ String addBeforeHandler =
+ p.context(BufferCallBeforeInitHandler.class).name();
+ p.addBefore(addBeforeHandler, "frameDecoder",
+ new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, 0, 4));
+ p.addBefore(addBeforeHandler, "rpcHandler", new RpcDuplexHandler(this));
+ p.fireUserEventTriggered(BufferCallEvent.success());
+ }
+
+ private Channel connect() {
+ if (channel != null) {
+ return channel;
+ }
+ channel = new Bootstrap().group(rpcClient.getGroup())
+ .channel(rpcClient.getChannelClass())
+ .option(ChannelOption.TCP_NODELAY, true)
+ .option(ChannelOption.SO_KEEPALIVE, true)
+ .handler(new BufferCallBeforeInitHandler())
+ .remoteAddress(remoteId.getAddress()).connect()
+ .addListener(new ChannelFutureListener() {
+
+ @Override
+ public void operationComplete(ChannelFuture future) throws Exception {
+ Channel ch = future.channel();
+ if (!future.isSuccess()) {
+ failInit(ch, IPCUtil.toIOE(future.cause()));
+ return;
+ }
+ writeConnectionHeader(ch);
+ writeConnectionContext(ch);
+ established(ch);
+ }
+ }).channel();
+ return channel;
+ }
+
+ private synchronized void failInit(Channel ch, IOException e) {
+ // fail all pending calls
+ ch.pipeline().fireUserEventTriggered(BufferCallEvent.fail(e));
+ shutdown0();
+ }
+
+ private void shutdown0() {
+ if (channel != null) {
+ channel.close();
+ channel = null;
+ }
+ }
+
+ public synchronized void shutdown() {
+ shutdown0();
+ }
+
+ public synchronized void sendRequest(Call call) {
+ Channel channel = connect();
+ channel.eventLoop().execute(() -> channel.writeAndFlush(call));
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcDuplexHandler.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcDuplexHandler.java
new file mode 100644
index 0000000..3cc5659
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ipc/RpcDuplexHandler.java
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.ipc.RPC.RpcKind;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.hadoop.ipc.protobuf.ProtobufRpcEngineProtos.RequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto.OperationProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto.RpcErrorCodeProto;
+import org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto.RpcStatusProto;
+import org.apache.hadoop.util.ProtoUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.protobuf.CodedOutputStream;
+import com.google.protobuf.Message;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufInputStream;
+import io.netty.buffer.ByteBufOutputStream;
+import io.netty.channel.ChannelDuplexHandler;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.ChannelPromise;
+
+@InterfaceAudience.Private
+class RpcDuplexHandler extends ChannelDuplexHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RpcDuplexHandler.class);
+
+ private final RpcConnection conn;
+
+ private final Map<Integer, Call> id2Call = new HashMap<>();
+
+ public RpcDuplexHandler(RpcConnection conn) {
+ this.conn = conn;
+ }
+
+ private void writeRequest(ChannelHandlerContext ctx, Call call,
+ ChannelPromise promise) throws IOException {
+ id2Call.put(call.getId(), call);
+
+ RpcRequestHeaderProto rpcHeader = ProtoUtil.makeRpcRequestHeader(
+ RpcKind.RPC_PROTOCOL_BUFFER, OperationProto.RPC_FINAL_PACKET,
+ call.getId(), 0, conn.rpcClient.getClientId());
+ int rpcHeaderSize = rpcHeader.getSerializedSize();
+ RequestHeaderProto requestHeader =
+ RequestHeaderProto.newBuilder().setMethodName(call.getMethodName())
+ .setDeclaringClassProtocolName(call.getProtocolName())
+ .setClientProtocolVersion(call.getProtocolVersion()).build();
+ int requestHeaderSize = requestHeader.getSerializedSize();
+ int totalSize = CodedOutputStream.computeRawVarint32Size(rpcHeaderSize) +
+ rpcHeaderSize +
+ CodedOutputStream.computeRawVarint32Size(requestHeaderSize) +
+ requestHeaderSize;
+ Message param = call.getParam();
+ if (param != null) {
+ int paramSize = param.getSerializedSize();
+ totalSize +=
+ CodedOutputStream.computeRawVarint32Size(paramSize) + paramSize;
+ }
+ ByteBufOutputStream out =
+ new ByteBufOutputStream(ctx.alloc().buffer(totalSize + 4));
+ out.writeInt(totalSize);
+ rpcHeader.writeDelimitedTo(out);
+ requestHeader.writeDelimitedTo(out);
+ if (param != null) {
+ param.writeDelimitedTo(out);
+ }
+ ctx.write(out.buffer(), promise);
+ }
+
+ @Override
+ public void write(ChannelHandlerContext ctx, Object msg,
+ ChannelPromise promise) throws Exception {
+ if (msg instanceof Call) {
+ writeRequest(ctx, (Call) msg, promise);
+ } else {
+ ctx.write(msg, promise);
+ }
+ }
+
+ private void readResponse(ChannelHandlerContext ctx, ByteBuf buf)
+ throws Exception {
+ ByteBufInputStream in = new ByteBufInputStream(buf);
+ RpcResponseHeaderProto header =
+ RpcResponseHeaderProto.parseDelimitedFrom(in);
+ int id = header.getCallId();
+ RpcStatusProto status = header.getStatus();
+ if (status != RpcStatusProto.SUCCESS) {
+ String exceptionClassName =
+ header.hasExceptionClassName() ? header.getExceptionClassName()
+ : "ServerDidNotSetExceptionClassName";
+ String errorMsg = header.hasErrorMsg() ? header.getErrorMsg()
+ : "ServerDidNotSetErrorMsg";
+ RpcErrorCodeProto errCode =
+ (header.hasErrorDetail() ? header.getErrorDetail() : null);
+ if (errCode == null) {
+ LOG.warn("Detailed error code not set by server on rpc error");
+ }
+ RemoteException re =
+ new RemoteException(exceptionClassName, errorMsg, errCode);
+ if (status == RpcStatusProto.ERROR) {
+ Call call = id2Call.remove(id);
+ call.setException(re);
+ } else if (status == RpcStatusProto.FATAL) {
+ exceptionCaught(ctx, re);
+ }
+ return;
+ }
+ Call call = id2Call.remove(id);
+ call.setResponse(call.getResponseDefaultType().getParserForType()
+ .parseDelimitedFrom(in));
+ }
+
+ @Override
+ public void channelRead(ChannelHandlerContext ctx, Object msg)
+ throws Exception {
+ if (msg instanceof ByteBuf) {
+ ByteBuf buf = (ByteBuf) msg;
+ try {
+ readResponse(ctx, buf);
+ } finally {
+ buf.release();
+ }
+ }
+ }
+
+ private void cleanupCalls(ChannelHandlerContext ctx, IOException error) {
+ for (Call call : id2Call.values()) {
+ call.setException(error);
+ }
+ id2Call.clear();
+ }
+
+ @Override
+ public void channelInactive(ChannelHandlerContext ctx) throws Exception {
+ if (!id2Call.isEmpty()) {
+ cleanupCalls(ctx, new IOException("Connection closed"));
+ }
+ conn.shutdown();
+ ctx.fireChannelInactive();
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause)
+ throws Exception {
+ if (!id2Call.isEmpty()) {
+ cleanupCalls(ctx, new IOException("Connection closed"));
+ }
+ conn.shutdown();
+ }
+}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestAsyncIPC.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestAsyncIPC.java
new file mode 100644
index 0000000..86fde48
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestAsyncIPC.java
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.protobuf.RpcChannel;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos;
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos.EchoRequestProto;
+import org.apache.hadoop.ipc.ProtobufRpcEngine;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestAsyncIPC {
+
+ private static Configuration CONF;
+
+ private static TestServer SERVER;
+
+ private static int PORT;
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ CONF = new Configuration();
+ RPC.setProtocolEngine(CONF, TestRpcProtocolPB.class,
+ ProtobufRpcEngine.class);
+ SERVER = new TestServer(CONF);
+ SERVER.start();
+ PORT = SERVER.port();
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ SERVER.stop();
+ }
+
+ @Test
+ public void test() throws IOException, InterruptedException {
+ try (RpcClient client = new RpcClient()) {
+ RpcChannel channel = client.createRpcChannel(TestRpcProtocolPB.class,
+ new InetSocketAddress("localhost", PORT),
+ UserGroupInformation.getCurrentUser());
+ TestRpcProtos.TestRpcService.Interface stub =
+ TestRpcProtos.TestRpcService.newStub(channel);
+ Map<Integer, String> results = new HashMap<>();
+ int count = 100;
+ CountDownLatch latch = new CountDownLatch(count);
+ for (int i = 0; i < count; i++) {
+ final int index = i;
+ stub.echo(new HdfsRpcController(),
+ EchoRequestProto.newBuilder().setMessage("Echo-" + index).build(),
+ resp -> {
+ results.put(index, resp.getMessage());
+ latch.countDown();
+ });
+ }
+ latch.await();
+ assertEquals(count, results.size());
+ for (int i = 0; i < count; i++) {
+ assertEquals("Echo-" + i, results.get(i));
+ }
+ }
+ }
+}
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestRpcProtocolPB.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestRpcProtocolPB.java
new file mode 100644
index 0000000..c7f7f27
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestRpcProtocolPB.java
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos;
+import org.apache.hadoop.ipc.ProtocolInfo;
+
+@ProtocolInfo(protocolName = "org.apache.hadoop.hdfs.ipc.TestRpcProtocol",
+ protocolVersion = 1)
+public interface TestRpcProtocolPB
+ extends TestRpcProtos.TestRpcService.BlockingInterface {
+}
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestServer.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestServer.java
new file mode 100644
index 0000000..3e06cc8
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/ipc/TestServer.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.ipc;
+
+import com.google.protobuf.RpcController;
+import com.google.protobuf.ServiceException;
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos;
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos.EchoRequestProto;
+import org.apache.hadoop.hdfs.ipc.protobuf.TestRpcProtos.EchoResponseProto;
+import org.apache.hadoop.ipc.RPC;
+
+public class TestServer implements TestRpcProtocolPB {
+
+ private final RPC.Server server;
+
+ public TestServer(Configuration conf) throws IOException {
+ server = new RPC.Builder(conf).setProtocol(TestRpcProtocolPB.class)
+ .setInstance(
+ TestRpcProtos.TestRpcService.newReflectiveBlockingService(this))
+ .setNumHandlers(10).build();
+ }
+
+ public void start() {
+ server.start();
+ }
+
+ public void stop() {
+ server.stop();
+ }
+
+ public int port() {
+ return server.getPort();
+ }
+
+ @Override
+ public EchoResponseProto echo(RpcController controller,
+ EchoRequestProto request) throws ServiceException {
+ return EchoResponseProto.newBuilder().setMessage(request.getMessage())
+ .build();
+ }
+}
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/test/proto/test_rpc.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/proto/test_rpc.proto
new file mode 100644
index 0000000..0997f56
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/test/proto/test_rpc.proto
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+option java_package = "org.apache.hadoop.hdfs.ipc.protobuf";
+option java_outer_classname = "TestRpcProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+package hadoop.hdfs;
+
+message EchoRequestProto {
+ required string message = 1;
+}
+
+message EchoResponseProto {
+ required string message = 1;
+}
+
+service TestRpcService {
+ rpc echo(EchoRequestProto) returns (EchoResponseProto);
+}
\ No newline at end of file
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 15c73b5..9be05d3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -842,7 +842,7 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
- <version>4.0.52.Final</version>
+ <version>4.1.37.Final</version>
</dependency>
<dependency>
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
index ba70199..8e4ef06 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
@@ -29,7 +29,6 @@
<protobuf.version>3.6.1</protobuf.version>
<guava.version>20.0</guava.version>
<grpc.version>1.15.1</grpc.version>
- <netty-all.version>4.1.27.Final</netty-all.version>
<os-maven-plugin.version>1.5.0.Final</os-maven-plugin.version>
</properties>
@@ -47,7 +46,6 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
- <version>${netty-all.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org