You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-commits@hadoop.apache.org by ae...@apache.org on 2019/10/25 17:36:31 UTC
[hadoop-ozone] branch HDDS-1880-Decom updated (f4e3fd1 -> a74cbfa)
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a change to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from f4e3fd1 HDDS-2349. QueryNode does not respect null values for opState or state
add f59d359 HDDS-2298. Fix maven warning about duplicated metrics-core jar
add 920dde9 HDDS-2298. Fix maven warning about duplicated metrics-core jar.
add e227ba4 HDDS-2220. HddsVolume needs a toString method.
add 83b5a67 HDDS-2220. HddsVolume needs a toString method.
add b3322b5 HDDS-2204. Avoid buffer coping in checksum verification. Contributed by Tsz Wo Nicholas Sze.
add 44a6a3f BlockManager should allocate a block in excluded pipelines if none other left (#19)
add 8868d4e HDDS-2194. Replication of Container fails with "Only closed containers could be exported" (#25)
add 546e36b HDDS-2305. Update Ozone to latest ratis snapshot(0.5.0-3f446aa-SNAPSHOT). Contributed by Mukul Kumar Singh. (#26)
add 616d7a4 HDDS-2295. Display log of freon on the standard output
add b4a1afd HDDS 2181. Ozone Manager should send correct ACL type in ACL requests to Authorizer Contributed by Vivek Ratnavel Subramanian.
add 59d66bc HDDS-1998. TestSecureContainerServer#testClientServerRatisGrpc is failing
add f1835dd Revert "HDDS-1998. TestSecureContainerServer#testClientServerRatisGrpc is failing"
add 5adf6a1 HDDS-1988. Fix listParts API
add fecdba7 HDDS-2267. Container metadata scanner interval mismatch
add 34df0bf HDDS-2312. Fix typo in ozone command
add 17081c2 Revert "HDDS 2181. Ozone Manager should send correct ACL type in ACL requests to Authorizer"
add 96f1dcc HDDS-1737. Add Volume check in KeyManager and File Operations
add 01e24e8 HDDS-2316. Support to skip recon and/or ozonefs during the build (#39)
add 0f5f7ba HDDS-2289. Put testing information and a problem description to the g… (#5)
add 9f88f42 HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions
add 61c161c Revert "HDDS-2032. Ozone client should retry writes in case of any ratis/stateMachine exceptions"
add 85ba643 HDDS-2302. Manage common pom versions in one common place.
add 38ec79e HDDS-2271. Avoid buffer copying in KeyValueHandler.
add 47f0bf5 Merge pull request #44 from szetszwo/HDDS-2271
add 6c575f2 HDDS-2275. In BatchOperation.SingleOperation, do not clone byte[].
add 61f4aa3 Merge pull request #45 from szetszwo/HDDS-2275
add b041050 HDDS-1985. Fix listVolumes API
add 37f48c4 HDDS-2221. Monitor datanodes in ozoneperf compose cluster
add bcd04c2 HDDS-2181. Ozone Manager should send correct ACL type in ACL requests to Authorizer (#43)
add 2529cee HDDS-2254 : Fix flaky unit test TestContainerStateMachine#testRatisSnapshotRetention.
add b961883 HDDS-2323. Mem allocation: Optimise AuditMessage::build(). Contributed by Siddharth Wagle.
add 25cfd0c HDDS-2318. Avoid proto::tostring in preconditions to save CPU cycles. (#48)
add 7651aec HDDS-2286. Add a log info in ozone client and scm to print the exclusion list during allocate block. Contributed by Siddharth Wagle. (#46)
add 18c4cab HDDS-2283. Container creation on datanodes take time because of Rocksdb option creation. Contributed by Siddharth Wagle.(#41)
add bfaa640 HDDS-2281. ContainerStateMachine#handleWriteChunk should ignore close container exception. Contributed by Shashikant Banerjee. (#54)
add 29cd37d HDDS-2280. HddsUtils#CheckForException should not return null in case the ratis exception cause is not set. Contributed by Shashikant Banerjee. (#57)
add aaa333c HDDS-2336. Fix TestKeyValueContainer#testRocksDBCreateUsesCachedOptions. Contributed by Attila Doroszlai. (#63)
add 65a1d00 HDDS-2337. Fix checkstyle errors. Contributed by Attila Doroszlai. (#64)
add 40f57a5 HDDS-2335. Params not included in AuditMessage. Contributed by Attila Doroszlai. (#62)
add 636df43 HDDS-2326. Http server of Freon is not started for new Freon tests (#52)
add 445fe62 HDDS-2206. Separate handling for OMException and IOException in the Ozone Manager. Contributed by Supratim Deka (#12)
add 02d2eb7 HDDS-2340. Updated ratis.version to get latest snapshot. (#67)
add e61e71d HDDS-2320. Negative value seen for OM NumKeys Metric in JMX. (#68)
add d47fe4e HDDS-2310. Add support to add ozone ranger plugin to Ozone Manager cl… (#49)
add c6c9794 HDDS-2333. Enable sync option for OM non-HA. (#61)
add 599b0d9 HDDS-2343. Add immutable entries in to the DoubleBuffer for Bucket requests. (#69)
add eb1d77e HDDS-2330. Random key generator can get stuck (#53)
new 9aa2282 Merge branch 'master' into HDDS-1880-Decom
new a74cbfa Merge branch 'HDDS-1880-Decom' of https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/pull_request_template.md | 18 +-
.../hadoop/hdds/scm/client/HddsClientUtils.java | 5 +-
.../scm/container/common/helpers/ExcludeList.java | 14 ++
.../hdds/scm/storage/ContainerProtocolCalls.java | 2 +-
.../apache/hadoop/hdds/utils/BatchOperation.java | 22 +--
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 59 +++---
.../org/apache/hadoop/hdds/utils/RocksDBStore.java | 11 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 21 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 8 +-
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 16 ++
.../apache/hadoop/ozone/audit/AuditMessage.java | 7 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 218 ++++++++++-----------
.../apache/hadoop/ozone/common/ChecksumData.java | 4 +-
.../ozone/container/common/helpers/BlockData.java | 5 +
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 29 ++-
.../apache/hadoop/ozone/common/TestChecksum.java | 6 +-
.../server/ratis/ContainerStateMachine.java | 12 +-
.../ozone/container/common/volume/HddsVolume.java | 8 +
.../container/keyvalue/KeyValueContainer.java | 13 +-
.../container/keyvalue/KeyValueContainerCheck.java | 3 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 9 +-
.../container/keyvalue/helpers/SmallFileUtils.java | 4 +-
.../ozoneimpl/ContainerMetadataScanner.java | 7 +-
.../container/keyvalue/TestKeyValueContainer.java | 53 +++--
hadoop-hdds/pom.xml | 8 -
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 12 ++
.../hdds/scm/server/SCMBlockProtocolServer.java | 2 +
.../hadoop/hdds/scm/block/TestBlockManager.java | 58 ++++--
.../client/io/BlockOutputStreamEntryPool.java | 1 +
.../ozone/client/protocol/ClientProtocol.java | 8 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 19 ++
hadoop-ozone/common/src/main/bin/ozone | 9 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 25 ++-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 32 ++-
...OzoneManagerProtocolClientSideTranslatorPB.java | 2 +-
.../ozone/security/acl/IAccessAuthorizer.java | 2 +-
.../src/main/proto/OzoneManagerProtocol.proto | 6 +
.../main/shellprofile.d/hadoop-ozone-manager.sh | 19 +-
hadoop-ozone/dev-support/checks/acceptance.sh | 2 +-
hadoop-ozone/dist/pom.xml | 90 +++++++--
.../src/main/compose/ozoneperf/docker-compose.yaml | 2 +-
.../prometheus => ozoneperf}/prometheus.yml | 10 +
.../client/rpc/TestContainerStateMachine.java | 26 +--
.../rpc/TestContainerStateMachineFailures.java | 118 +++++++++++
.../ozone/container/ContainerTestHelper.java | 25 ++-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 6 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 135 +++++++++++++
.../security/acl/TestOzoneNativeAuthorizer.java | 23 ++-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 37 ++--
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 22 ++-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 49 +++++
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 19 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 3 +-
.../om/request/bucket/OMBucketCreateRequest.java | 5 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 2 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 5 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 3 +-
.../om/request/file/OMDirectoryCreateRequest.java | 22 +--
.../ozone/om/request/file/OMFileCreateRequest.java | 16 +-
.../om/request/key/OMAllocateBlockRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 17 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 5 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 16 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 16 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 39 +++-
.../OzoneManagerHARequestHandlerImpl.java | 61 +-----
...OzoneManagerProtocolServerSideTranslatorPB.java | 21 +-
.../ozone/security/acl/OzoneNativeAuthorizer.java | 52 +++--
.../request/file/TestOMDirectoryCreateRequest.java | 73 ++++++-
.../om/request/file/TestOMFileCreateRequest.java | 21 ++
.../om/request/key/TestOMKeyDeleteRequest.java | 26 ++-
.../om/request/key/TestOMKeyRenameRequest.java | 28 ++-
.../ozone/om/request/key/TestOMKeyRequest.java | 1 +
hadoop-ozone/pom.xml | 38 ++--
hadoop-ozone/tools/pom.xml | 5 -
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 68 +++++++
.../ozone/admin/om/GetServiceRolesSubcommand.java} | 47 ++---
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 46 +++--
.../hadoop/ozone/admin/om}/package-info.java | 5 +-
.../apache/hadoop/ozone/admin}/package-info.java | 5 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 11 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 12 +-
pom.xml | 15 +-
83 files changed, 1456 insertions(+), 553 deletions(-)
copy hadoop-hdds/docs/dev-support/bin/generate-site.sh => hadoop-ozone/common/src/main/shellprofile.d/hadoop-ozone-manager.sh (69%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/compose/{common/prometheus => ozoneperf}/prometheus.yml (72%)
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/OzoneAdmin.java
copy hadoop-ozone/{insight/src/main/java/org/apache/hadoop/ozone/insight/ListSubCommand.java => tools/src/main/java/org/apache/hadoop/ozone/admin/om/GetServiceRolesSubcommand.java} (54%)
copy hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCommands.java => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java (54%)
copy {hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om}/package-info.java (92%)
copy {hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli => hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin}/package-info.java (92%)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-commits-help@hadoop.apache.org
[hadoop-ozone] 01/02: Merge branch 'master' into HDDS-1880-Decom
Posted by ae...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a commit to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit 9aa228255873a169ebcb896408157a23a1950863
Merge: 4f58c81 eb1d77e
Author: Anu Engineer <ae...@apache.org>
AuthorDate: Fri Oct 25 10:32:00 2019 -0700
Merge branch 'master' into HDDS-1880-Decom
Some tests are failing:
org.apache.hadoop.hdds.scm.block.TestBlockManager
org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
.github/pull_request_template.md | 18 +-
.../hadoop/hdds/scm/client/HddsClientUtils.java | 5 +-
.../scm/container/common/helpers/ExcludeList.java | 14 ++
.../hdds/scm/storage/ContainerProtocolCalls.java | 2 +-
.../apache/hadoop/hdds/utils/BatchOperation.java | 22 +--
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 59 +++---
.../org/apache/hadoop/hdds/utils/RocksDBStore.java | 11 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 21 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 8 +-
.../hadoop/hdds/utils/db/RocksDBConfiguration.java | 16 ++
.../apache/hadoop/ozone/audit/AuditMessage.java | 7 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 218 ++++++++++-----------
.../apache/hadoop/ozone/common/ChecksumData.java | 4 +-
.../ozone/container/common/helpers/BlockData.java | 5 +
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 29 ++-
.../apache/hadoop/ozone/common/TestChecksum.java | 6 +-
.../server/ratis/ContainerStateMachine.java | 12 +-
.../ozone/container/common/volume/HddsVolume.java | 8 +
.../container/keyvalue/KeyValueContainer.java | 13 +-
.../container/keyvalue/KeyValueContainerCheck.java | 3 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 9 +-
.../container/keyvalue/helpers/SmallFileUtils.java | 4 +-
.../ozoneimpl/ContainerMetadataScanner.java | 7 +-
.../container/keyvalue/TestKeyValueContainer.java | 53 +++--
hadoop-hdds/pom.xml | 8 -
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 12 ++
.../hdds/scm/server/SCMBlockProtocolServer.java | 2 +
.../hadoop/hdds/scm/block/TestBlockManager.java | 58 ++++--
.../client/io/BlockOutputStreamEntryPool.java | 1 +
.../ozone/client/protocol/ClientProtocol.java | 8 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 19 ++
hadoop-ozone/common/src/main/bin/ozone | 9 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 25 ++-
.../hadoop/ozone/om/helpers/ServiceInfo.java | 32 ++-
...OzoneManagerProtocolClientSideTranslatorPB.java | 2 +-
.../ozone/security/acl/IAccessAuthorizer.java | 2 +-
.../src/main/proto/OzoneManagerProtocol.proto | 6 +
.../main/shellprofile.d/hadoop-ozone-manager.sh} | 25 +--
hadoop-ozone/dev-support/checks/acceptance.sh | 2 +-
hadoop-ozone/dist/pom.xml | 90 +++++++--
.../src/main/compose/ozoneperf/docker-compose.yaml | 2 +-
.../prometheus => ozoneperf}/prometheus.yml | 10 +
.../client/rpc/TestContainerStateMachine.java | 26 +--
.../rpc/TestContainerStateMachineFailures.java | 118 +++++++++++
.../ozone/container/ContainerTestHelper.java | 25 ++-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 6 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 135 +++++++++++++
.../security/acl/TestOzoneNativeAuthorizer.java | 23 ++-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 37 ++--
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 22 ++-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 49 +++++
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 19 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 3 +-
.../om/request/bucket/OMBucketCreateRequest.java | 5 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 2 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 5 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 3 +-
.../om/request/file/OMDirectoryCreateRequest.java | 22 +--
.../ozone/om/request/file/OMFileCreateRequest.java | 16 +-
.../om/request/key/OMAllocateBlockRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 17 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 5 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 16 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 16 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 39 +++-
.../OzoneManagerHARequestHandlerImpl.java | 61 +-----
...OzoneManagerProtocolServerSideTranslatorPB.java | 21 +-
.../ozone/security/acl/OzoneNativeAuthorizer.java | 52 +++--
.../request/file/TestOMDirectoryCreateRequest.java | 73 ++++++-
.../om/request/file/TestOMFileCreateRequest.java | 21 ++
.../om/request/key/TestOMKeyDeleteRequest.java | 26 ++-
.../om/request/key/TestOMKeyRenameRequest.java | 28 ++-
.../ozone/om/request/key/TestOMKeyRequest.java | 1 +
hadoop-ozone/pom.xml | 38 ++--
hadoop-ozone/tools/pom.xml | 5 -
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 68 +++++++
.../ozone/admin/om/GetServiceRolesSubcommand.java | 60 ++++++
.../org/apache/hadoop/ozone/admin/om/OMAdmin.java | 63 ++++++
.../apache/hadoop/ozone/admin/om/package-info.java | 24 +++
.../apache/hadoop/ozone/admin/package-info.java | 24 +++
.../hadoop/ozone/freon/BaseFreonGenerator.java | 11 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 12 +-
pom.xml | 15 +-
83 files changed, 1572 insertions(+), 511 deletions(-)
diff --cc hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java
index 1b8f5bb,1b8f5bb..3773915
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java
@@@ -42,6 -42,6 +42,7 @@@ import java.util.Map
import org.apache.hadoop.ozone.om.helpers.OzoneFileStatus;
import org.apache.hadoop.ozone.om.helpers.S3SecretValue;
++import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
import org.apache.hadoop.ozone.security.OzoneTokenIdentifier;
import org.apache.hadoop.ozone.security.acl.OzoneObj;
import org.apache.hadoop.security.KerberosInfo;
@@@ -59,6 -59,6 +60,13 @@@ import org.apache.hadoop.security.token
public interface ClientProtocol {
/**
++ * List of OM node Ids and their Ratis server roles.
++ * @return List of OM server roles
++ * @throws IOException
++ */
++ List<OMRoleInfo> getOmRoleInfos() throws IOException;
++
++ /**
* Creates a new Volume.
* @param volumeName Name of the Volume
* @throws IOException
diff --cc hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
index 06351ab,06351ab..a9d221b
--- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
+++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
@@@ -64,6 -64,6 +64,7 @@@ import org.apache.hadoop.ozone.om.helpe
import org.apache.hadoop.ozone.om.helpers.OzoneAclUtil;
import org.apache.hadoop.ozone.om.helpers.OzoneFileStatus;
import org.apache.hadoop.ozone.om.helpers.S3SecretValue;
++import org.apache.hadoop.ozone.om.helpers.ServiceInfo;
import org.apache.hadoop.ozone.om.helpers.ServiceInfoEx;
import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
import org.apache.hadoop.ozone.om.protocolPB
@@@ -72,6 -72,6 +73,7 @@@ import org.apache.hadoop.ozone.OzoneAcl
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.XceiverClientManager;
++import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
import org.apache.hadoop.ozone.security.GDPRSymmetricKey;
import org.apache.hadoop.ozone.security.OzoneTokenIdentifier;
import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer.ACLIdentityType;
@@@ -224,6 -224,6 +226,23 @@@ public class RpcClient implements Clien
}
@Override
++ public List<OMRoleInfo> getOmRoleInfos() throws IOException {
++
++ List<ServiceInfo> serviceList = ozoneManagerClient.getServiceList();
++ List<OMRoleInfo> roleInfos = new ArrayList<>();
++
++ for (ServiceInfo serviceInfo : serviceList) {
++ if (serviceInfo.getNodeType().equals(HddsProtos.NodeType.OM)) {
++ OMRoleInfo omRoleInfo = serviceInfo.getOmRoleInfo();
++ if (omRoleInfo != null) {
++ roleInfos.add(omRoleInfo);
++ }
++ }
++ }
++ return roleInfos;
++ }
++
++ @Override
public void createVolume(String volumeName) throws IOException {
createVolume(volumeName, VolumeArgs.newBuilder().build());
}
diff --cc hadoop-ozone/common/src/main/bin/ozone
index cd8f202,9c13b94..987e118
--- a/hadoop-ozone/common/src/main/bin/ozone
+++ b/hadoop-ozone/common/src/main/bin/ozone
@@@ -55,6 -55,6 +55,7 @@@ function hadoop_usag
hadoop_add_subcommand "version" client "print the version"
hadoop_add_subcommand "dtutil" client "operations related to delegation tokens"
hadoop_add_subcommand "upgrade" client "HDFS to Ozone in-place upgrade tool"
++ hadoop_add_subcommand "admin" client "Ozone admin tool"
hadoop_generate_usage "${HADOOP_SHELL_EXECNAME}" false
}
@@@ -207,6 -207,6 +208,10 @@@ function ozonecmd_cas
HADOOP_CLASSNAME=org.apache.hadoop.ozone.upgrade.InPlaceUpgrade
OZONE_RUN_ARTIFACT_NAME="hadoop-ozone-upgrade"
;;
++ admin)
++ HADOOP_CLASSNAME=org.apache.hadoop.ozone.admin.OzoneAdmin
++ OZONE_RUN_ARTIFACT_NAME="hadoop-ozone-tools"
++ ;;
*)
HADOOP_CLASSNAME="${subcmd}"
if ! hadoop_validate_classname "${HADOOP_CLASSNAME}"; then
diff --cc hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java
index dce4f8e,dce4f8e..e569db8
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java
@@@ -26,6 -26,6 +26,8 @@@ import com.fasterxml.jackson.databind.O
import com.google.common.base.Preconditions;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
++ .OMRoleInfo;
++import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
.ServicePort;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType;
@@@ -58,6 -58,6 +60,8 @@@ public final class ServiceInfo
* List of ports the service listens to.
*/
private Map<ServicePort.Type, Integer> ports;
++
++ private OMRoleInfo omRoleInfo;
/**
* Default constructor for JSON deserialization.
@@@ -70,8 -70,8 +74,8 @@@
* @param hostname hostname of the service
* @param portList list of ports the service listens to
*/
-- private ServiceInfo(
-- NodeType nodeType, String hostname, List<ServicePort> portList) {
++ private ServiceInfo(NodeType nodeType, String hostname,
++ List<ServicePort> portList, OMRoleInfo omRole) {
Preconditions.checkNotNull(nodeType);
Preconditions.checkNotNull(hostname);
this.nodeType = nodeType;
@@@ -80,6 -80,6 +84,7 @@@
for (ServicePort port : portList) {
ports.put(port.getType(), port.getValue());
}
++ this.omRoleInfo = omRole;
}
/**
@@@ -129,6 -129,6 +134,15 @@@
}
/**
++ * Returns the OM role info - node id and ratis server role.
++ * @return OmRoleInfo
++ */
++ @JsonIgnore
++ public OMRoleInfo getOmRoleInfo() {
++ return omRoleInfo;
++ }
++
++ /**
* Converts {@link ServiceInfo} to OzoneManagerProtocolProtos.ServiceInfo.
*
* @return OzoneManagerProtocolProtos.ServiceInfo
@@@ -147,6 -147,6 +161,9 @@@
.setType(entry.getKey())
.setValue(entry.getValue()).build())
.collect(Collectors.toList()));
++ if (nodeType == NodeType.OM && omRoleInfo != null) {
++ builder.setOmRole(omRoleInfo);
++ }
return builder.build();
}
@@@ -160,7 -160,7 +177,8 @@@
OzoneManagerProtocolProtos.ServiceInfo serviceInfo) {
return new ServiceInfo(serviceInfo.getNodeType(),
serviceInfo.getHostname(),
-- serviceInfo.getServicePortsList());
++ serviceInfo.getServicePortsList(),
++ serviceInfo.hasOmRole() ? serviceInfo.getOmRole() : null);
}
/**
@@@ -179,7 -179,7 +197,7 @@@
private NodeType node;
private String host;
private List<ServicePort> portList = new ArrayList<>();
--
++ private OMRoleInfo omRoleInfo;
/**
* Sets the node/service type.
@@@ -211,13 -211,13 +229,17 @@@
return this;
}
++ public Builder setOmRoleInfo(OMRoleInfo omRole) {
++ omRoleInfo = omRole;
++ return this;
++ }
/**
* Builds and returns {@link ServiceInfo} with the set values.
* @return {@link ServiceInfo}
*/
public ServiceInfo build() {
-- return new ServiceInfo(node, host, portList);
++ return new ServiceInfo(node, host, portList, omRoleInfo);
}
}
diff --cc hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
index d82fdf2,d82fdf2..b9ccdeb
--- a/hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
+++ b/hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
@@@ -903,10 -903,10 +903,16 @@@ message ServicePort
required uint32 value = 2;
}
++message OMRoleInfo {
++ required string nodeId = 1;
++ required string serverRole = 2;
++}
++
message ServiceInfo {
required hadoop.hdds.NodeType nodeType = 1;
required string hostname = 2;
repeated ServicePort servicePorts = 3;
++ optional OMRoleInfo omRole = 4;
}
message S3CreateBucketRequest {
diff --cc hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 902fe6a,294b74e..fbda8c8
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@@ -86,6 -86,6 +86,7 @@@ import org.apache.hadoop.ozone.protocol
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.DBUpdatesRequest;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
.KeyArgs;
++import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
import org.apache.hadoop.ozone.protocolPB.ProtocolMessageMetrics;
import org.apache.hadoop.ozone.security.OzoneSecurityException;
import org.apache.hadoop.ozone.security.OzoneTokenIdentifier;
@@@ -154,6 -154,6 +155,7 @@@ import org.apache.hadoop.hdds.utils.db.
import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
import org.apache.hadoop.hdds.utils.db.DBStore;
++import org.apache.ratis.proto.RaftProtos.RaftPeerRole;
import org.apache.ratis.server.protocol.TermIndex;
import org.apache.ratis.util.FileUtils;
import org.apache.ratis.util.LifeCycle;
@@@ -2381,6 -2384,6 +2386,43 @@@ public final class OzoneManager extend
.setValue(httpServer.getHttpsAddress().getPort())
.build());
}
++
++ // Since this OM is processing the request, we can assume it to be the
++ // leader OM
++
++ OMRoleInfo omRole = OMRoleInfo.newBuilder()
++ .setNodeId(getOMNodeId())
++ .setServerRole(RaftPeerRole.LEADER.name())
++ .build();
++ omServiceInfoBuilder.setOmRoleInfo(omRole);
++
++ if (isRatisEnabled) {
++ if (omRatisServer != null) {
++ omServiceInfoBuilder.addServicePort(ServicePort.newBuilder()
++ .setType(ServicePort.Type.RATIS)
++ .setValue(omNodeDetails.getRatisPort())
++ .build());
++ }
++
++ for (OMNodeDetails peerNode : peerNodes) {
++ ServiceInfo.Builder peerOmServiceInfoBuilder = ServiceInfo.newBuilder()
++ .setNodeType(HddsProtos.NodeType.OM)
++ .setHostname(peerNode.getAddress().getHostName())
++ .addServicePort(ServicePort.newBuilder()
++ .setType(ServicePort.Type.RPC)
++ .setValue(peerNode.getRpcPort())
++ .build());
++
++ OMRoleInfo peerOmRole = OMRoleInfo.newBuilder()
++ .setNodeId(peerNode.getOMNodeId())
++ .setServerRole(RaftPeerRole.FOLLOWER.name())
++ .build();
++ peerOmServiceInfoBuilder.setOmRoleInfo(peerOmRole);
++
++ services.add(peerOmServiceInfoBuilder.build());
++ }
++ }
++
services.add(omServiceInfoBuilder.build());
// For client we have to return SCM with container protocol port,
diff --cc hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/OzoneAdmin.java
index 0000000,0000000..27ef698
new file mode 100644
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/OzoneAdmin.java
@@@ -1,0 -1,0 +1,68 @@@
++/**
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ * <p>
++ * http://www.apache.org/licenses/LICENSE-2.0
++ * <p>
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
++package org.apache.hadoop.ozone.admin;
++
++import org.apache.hadoop.hdds.cli.GenericCli;
++import org.apache.hadoop.hdds.cli.HddsVersionProvider;
++import org.apache.hadoop.hdds.conf.OzoneConfiguration;
++import org.apache.hadoop.ozone.admin.om.OMAdmin;
++import org.apache.hadoop.util.NativeCodeLoader;
++import org.apache.log4j.ConsoleAppender;
++import org.apache.log4j.Level;
++import org.apache.log4j.LogManager;
++import org.apache.log4j.Logger;
++import org.apache.log4j.PatternLayout;
++import picocli.CommandLine;
++
++/**
++ * Ozone Admin Command line tool.
++ */
++@CommandLine.Command(name = "ozone admin",
++ hidden = true,
++ description = "Developer tools for Ozone Admin operations",
++ versionProvider = HddsVersionProvider.class,
++ subcommands = {
++ OMAdmin.class
++ },
++ mixinStandardHelpOptions = true)
++public class OzoneAdmin extends GenericCli {
++ private OzoneConfiguration ozoneConf;
++
++ public OzoneConfiguration getOzoneConf() {
++ if (ozoneConf == null) {
++ ozoneConf = createOzoneConfiguration();
++ }
++ return ozoneConf;
++ }
++
++ /**
++ * Main for the Ozone Admin shell Command handling.
++ *
++ * @param argv - System Args Strings[]
++ * @throws Exception
++ */
++ public static void main(String[] argv) throws Exception {
++ LogManager.resetConfiguration();
++ Logger.getRootLogger().setLevel(Level.INFO);
++ Logger.getRootLogger()
++ .addAppender(new ConsoleAppender(new PatternLayout("%m%n")));
++ Logger.getLogger(NativeCodeLoader.class).setLevel(Level.ERROR);
++
++ new OzoneAdmin().run(argv);
++ }
++}
diff --cc hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/GetServiceRolesSubcommand.java
index 0000000,0000000..2984936
new file mode 100644
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/GetServiceRolesSubcommand.java
@@@ -1,0 -1,0 +1,60 @@@
++/*
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ *
++ * http://www.apache.org/licenses/LICENSE-2.0
++ *
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
++
++package org.apache.hadoop.ozone.admin.om;
++
++import org.apache.hadoop.hdds.cli.HddsVersionProvider;
++import org.apache.hadoop.ozone.client.protocol.ClientProtocol;
++import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
++import picocli.CommandLine;
++
++import java.util.List;
++import java.util.concurrent.Callable;
++
++/**
++ * Handler of om get-service-roles command.
++ */
++@CommandLine.Command(
++ name = "getserviceroles",
++ description = "List all OMs and their respective Ratis server roles",
++ mixinStandardHelpOptions = true,
++ versionProvider = HddsVersionProvider.class)
++public class GetServiceRolesSubcommand implements Callable<Void> {
++
++ @CommandLine.ParentCommand
++ private OMAdmin parent;
++
++ @CommandLine.Option(names = {"-id", "--om-service-id"},
++ description = "OM Service ID",
++ required = true)
++ private String omServiceId;
++
++ @Override
++ public Void call() throws Exception {
++ ClientProtocol client = parent.createClient(omServiceId);
++ getOmServerRoles(client.getOmRoleInfos());
++ return null;
++ }
++
++ private void getOmServerRoles(List<OMRoleInfo> roleInfos) {
++ for (OMRoleInfo roleInfo : roleInfos) {
++ System.out.println(
++ roleInfo.getNodeId() + " : " + roleInfo.getServerRole());
++ }
++ }
++}
diff --cc hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java
index 0000000,0000000..3a109ac
new file mode 100644
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java
@@@ -1,0 -1,0 +1,63 @@@
++/**
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ * <p>
++ * http://www.apache.org/licenses/LICENSE-2.0
++ * <p>
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ */
++package org.apache.hadoop.ozone.admin.om;
++
++import org.apache.hadoop.hdds.cli.GenericCli;
++import org.apache.hadoop.hdds.cli.HddsVersionProvider;
++import org.apache.hadoop.hdds.cli.MissingSubcommandException;
++import org.apache.hadoop.hdds.conf.OzoneConfiguration;
++import org.apache.hadoop.ozone.admin.OzoneAdmin;
++import org.apache.hadoop.ozone.client.OzoneClientFactory;
++import org.apache.hadoop.ozone.client.protocol.ClientProtocol;
++import picocli.CommandLine;
++
++import java.io.IOException;
++
++/**
++ * Subcommand for admin operations related to OM.
++ */
++@CommandLine.Command(
++ name = "om",
++ description = "Ozone Manager specific admin operations",
++ mixinStandardHelpOptions = true,
++ versionProvider = HddsVersionProvider.class,
++ subcommands = {
++ GetServiceRolesSubcommand.class
++ })
++public class OMAdmin extends GenericCli {
++
++ @CommandLine.ParentCommand
++ private OzoneAdmin parent;
++
++ public OzoneAdmin getParent() {
++ return parent;
++ }
++
++ @Override
++ public Void call() throws Exception {
++ throw new MissingSubcommandException(
++ this.parent.getCmd().getSubcommands().get("om"));
++ }
++
++ public ClientProtocol createClient(String omServiceId) throws IOException {
++ OzoneConfiguration conf = parent.getOzoneConf();
++ return OzoneClientFactory.getRpcClient(omServiceId, conf).getObjectStore()
++ .getClientProxy();
++
++ }
++}
diff --cc hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/package-info.java
index 0000000,0000000..0fa52da
new file mode 100644
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/package-info.java
@@@ -1,0 -1,0 +1,24 @@@
++/**
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ * <p>
++ * http://www.apache.org/licenses/LICENSE-2.0
++ * <p>
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ * <p>
++ * SCM related cli tools.
++ */
++
++/**
++ * OM related Admin tools.
++ */
++package org.apache.hadoop.ozone.admin.om;
diff --cc hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/package-info.java
index 0000000,0000000..3a30a38
new file mode 100644
--- /dev/null
+++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/package-info.java
@@@ -1,0 -1,0 +1,24 @@@
++/**
++ * Licensed to the Apache Software Foundation (ASF) under one
++ * or more contributor license agreements. See the NOTICE file
++ * distributed with this work for additional information
++ * regarding copyright ownership. The ASF licenses this file
++ * to you under the Apache License, Version 2.0 (the
++ * "License"); you may not use this file except in compliance
++ * with the License. You may obtain a copy of the License at
++ * <p>
++ * http://www.apache.org/licenses/LICENSE-2.0
++ * <p>
++ * Unless required by applicable law or agreed to in writing, software
++ * distributed under the License is distributed on an "AS IS" BASIS,
++ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++ * See the License for the specific language governing permissions and
++ * limitations under the License.
++ * <p>
++ * SCM related cli tools.
++ */
++
++/**
++ * Ozone Admin tools.
++ */
++package org.apache.hadoop.ozone.admin;
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-commits-help@hadoop.apache.org
[hadoop-ozone] 02/02: Merge branch 'HDDS-1880-Decom' of
https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
Posted by ae...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a commit to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
commit a74cbfa1c6a302885096a9d71afd5e2677f2bf91
Merge: 9aa2282 f4e3fd1
Author: Anu Engineer <ae...@apache.org>
AuthorDate: Fri Oct 25 10:32:39 2019 -0700
Merge branch 'HDDS-1880-Decom' of https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
...ontainerLocationProtocolServerSideTranslatorPB.java | 10 ++++++++--
.../hdds/scm/server/SCMClientProtocolServer.java | 18 +++++++++---------
.../apache/hadoop/ozone/scm/node/TestQueryNode.java | 12 +++++++++---
3 files changed, 26 insertions(+), 14 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-commits-help@hadoop.apache.org