You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by zh...@apache.org on 2019/11/10 13:26:10 UTC
[hbase] branch HBASE-22514 updated (4a2251f -> eb83d58)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a change to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git.
discard 4a2251f HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
discard 9142557 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
discard 4f6d20a HBASE-23232 Remove rsgroup profile from pom.xml of hbase-assembly (#779)
discard 2584c39 HBASE-23081 Add an option to enable/disable rs group feature (#691)
discard f989e8a HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
discard 46d922a HBASE-22987 Calculate the region servers in default group in foreground (#599)
discard 39911b4 HBASE-22729 Start RSGroupInfoManager as default (#555)
discard c162c88 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
discard 935f0ac HBASE-22820 Do not need to persist default rs group now (#482)
discard 225ee08 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
discard c143757 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
discard 6a35615 HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
discard 22c67b4 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
discard 2e02c6c HBASE-22664 Move protobuf stuff in hbase-rsgroup to hbase-protocol-shaded (#362)
add 56e63b1 HBASE-23250 Log message about CleanerChore delegate initialization should be at INFO
add f0f7fae HBASE-21458 Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
add d2142a8 HBASE-23085 Network and Data related Actions
add f58bd4a HBASE-22980 HRegionPartioner getPartition() method incorrectly partitions the regions of the table. (#590)
add 12c19a6 HBASE-23230 Enforce member visibility in HRegionServer (#775)
add 34d5b3b HBASE-23272 Fix link in Developer guide to "code review checklist" (#805)
add 02f2643 HBASE-22480 Get block from BlockCache once and return this block to BlockCache twice make ref count error.
add c348e64 HBASE-23262 Cannot load Master UI (#799)
add 264d3e3 HBASE-23263 NPE in Quotas.jsp (#800)
add c2e01f2 HBASE-23257: Track clusterID in stand by masters (#798)
add b53e9a3 HBASE-23236 Upgrade to yetus 0.11.1
add 3466105 HBASE-22888 Share some stuffs with the initial reader when new stream reader created (#581)
add 2c1d500 HBASE-23268 Remove disable/enable ops from doc when altering schema
add d5dcc09 HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804)
add 08aae42 HBASE-18439 Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
new 13df3a0 HBASE-22664 Move protobuf stuff in hbase-rsgroup to hbase-protocol-shaded (#362)
new da62506 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
new 6aede3f HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
new ed2d5b9 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
new 83fa1e4 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
new 88e3610 HBASE-22820 Do not need to persist default rs group now (#482)
new 3dc1b28 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
new 90d4fb7 HBASE-22729 Start RSGroupInfoManager as default (#555)
new cf5295a HBASE-22987 Calculate the region servers in default group in foreground (#599)
new 879b08c HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
new 5d9620d HBASE-23081 Add an option to enable/disable rs group feature (#691)
new 0201d4f HBASE-23232 Remove rsgroup profile from pom.xml of hbase-assembly (#779)
new c1480c0 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
new eb83d58 HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4a2251f)
\
N -- N -- N refs/heads/HBASE-22514 (eb83d58)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 14 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/hbase | 23 +-
dev-support/Jenkinsfile | 2 +-
dev-support/Jenkinsfile_GitHub | 2 +-
dev-support/hbase-personality.sh | 20 +-
.../hadoop/hbase/backup/util/RestoreTool.java | 1 -
.../apache/hadoop/hbase/HBaseClusterManager.java | 90 ++-
.../apache/hadoop/hbase/chaos/actions/Action.java | 11 +-
.../hbase/chaos/actions/AddCPULoadAction.java | 69 +++
.../hbase/chaos/actions/AddColumnAction.java | 3 +
.../hbase/chaos/actions/BatchRestartRsAction.java | 5 +
.../chaos/actions/ChangeBloomFilterAction.java | 3 +
.../chaos/actions/ChangeCompressionAction.java | 9 +-
.../hbase/chaos/actions/ChangeEncodingAction.java | 9 +-
.../chaos/actions/ChangeSplitPolicyAction.java | 7 +-
.../hbase/chaos/actions/ChangeVersionsAction.java | 5 +-
.../hbase/chaos/actions/CompactMobAction.java | 3 +
.../actions/CompactRandomRegionOfTableAction.java | 5 +
.../hbase/chaos/actions/CompactTableAction.java | 3 +
.../chaos/actions/CorruptDataFilesAction.java | 75 +++
.../actions/CorruptPackagesCommandAction.java | 72 +++
.../chaos/actions/DelayPackagesCommandAction.java | 71 +++
.../hbase/chaos/actions/DeleteDataFilesAction.java | 66 +++
.../chaos/actions/DumpClusterStatusAction.java | 5 +
.../actions/DuplicatePackagesCommandAction.java | 72 +++
.../hbase/chaos/actions/FillDiskCommandAction.java | 83 +++
.../actions/FlushRandomRegionOfTableAction.java | 6 +-
.../hbase/chaos/actions/FlushTableAction.java | 4 +
.../hbase/chaos/actions/ForceBalancerAction.java | 6 +
.../chaos/actions/LosePackagesCommandAction.java | 72 +++
.../MergeRandomAdjacentRegionsOfTableAction.java | 4 +
.../actions/MoveRandomRegionOfTableAction.java | 4 +
.../chaos/actions/MoveRegionsOfTableAction.java | 5 +
.../hbase/chaos/actions/RemoveColumnAction.java | 4 +
.../actions/ReorderPackagesCommandAction.java | 76 +++
.../chaos/actions/RestartActionBaseAction.java | 4 +
.../chaos/actions/RestartActiveMasterAction.java | 4 +
.../chaos/actions/RestartActiveNameNodeAction.java | 4 +
.../chaos/actions/RestartRandomDataNodeAction.java | 12 +-
.../hbase/chaos/actions/RestartRandomRsAction.java | 4 +
.../chaos/actions/RestartRandomZKNodeAction.java | 4 +
.../chaos/actions/RestartRsHoldingMetaAction.java | 5 +
.../chaos/actions/RestartRsHoldingTableAction.java | 4 +
.../chaos/actions/RollingBatchRestartRsAction.java | 47 +-
.../hbase/chaos/actions/SnapshotTableAction.java | 4 +
.../chaos/actions/SplitAllRegionOfTableAction.java | 10 +-
.../actions/SplitRandomRegionOfTableAction.java | 5 +
.../hbase/chaos/actions/SudoCommandAction.java | 70 +++
...rceBalancerAction.java => TCCommandAction.java} | 21 +-
.../hbase/chaos/actions/TruncateTableAction.java | 6 +-
.../actions/UnbalanceKillAndRebalanceAction.java | 5 +
.../chaos/actions/UnbalanceRegionsAction.java | 5 +
...eyFactory.java => DataIssuesMonkeyFactory.java} | 43 +-
.../factories/DistributedIssuesMonkeyFactory.java | 121 ++++
.../hbase/chaos/factories/MonkeyConstants.java | 29 +-
.../hbase/chaos/factories/MonkeyFactory.java | 4 +
.../ServerAndDependenciesKillingMonkeyFactory.java | 8 +-
.../factories/ServerKillingMonkeyFactory.java | 8 +-
.../factories/SlowDeterministicMonkeyFactory.java | 8 +-
.../StressAssignmentManagerMonkeyFactory.java | 12 +-
.../hadoop/hbase/mapred/HRegionPartitioner.java | 2 +-
.../hadoop/hbase/mapreduce/HFileInputFormat.java | 1 -
.../hadoop/hbase/mapreduce/HRegionPartitioner.java | 2 +-
.../hbase/mapreduce/TestHFileOutputFormat2.java | 7 +-
.../hbase/mapreduce/TestHRegionPartitioner.java | 24 +
.../TestImportTSVWithVisibilityLabels.java | 1 -
.../hadoop/hbase/mapreduce/TestImportTsv.java | 1 -
.../hbase/tmpl/master/MasterStatusTmpl.jamon | 3 +-
.../hbase/tmpl/regionserver/RSStatusTmpl.jamon | 2 +-
.../main/java/org/apache/hadoop/hbase/Server.java | 3 +-
.../hadoop/hbase/io/HalfStoreFileReader.java | 54 +-
.../org/apache/hadoop/hbase/io/hfile/HFile.java | 376 ++-----------
.../apache/hadoop/hbase/io/hfile/HFileBlock.java | 24 +-
.../hadoop/hbase/io/hfile/HFileBlockIndex.java | 44 +-
.../hbase/io/hfile/HFileDataBlockEncoderImpl.java | 5 +-
.../apache/hadoop/hbase/io/hfile/HFileInfo.java | 503 +++++++++++++++++
.../hadoop/hbase/io/hfile/HFilePreadReader.java | 109 ++++
.../hadoop/hbase/io/hfile/HFilePrettyPrinter.java | 38 +-
.../hadoop/hbase/io/hfile/HFileReaderImpl.java | 430 ++++----------
...dHFileException.java => HFileStreamReader.java} | 24 +-
.../hadoop/hbase/io/hfile/HFileWriterImpl.java | 28 +-
.../hadoop/hbase/io/hfile/ReaderContext.java | 76 +++
.../hbase/io/hfile/ReaderContextBuilder.java | 106 ++++
.../hadoop/hbase/master/CachedClusterId.java | 155 +++++
.../org/apache/hadoop/hbase/master/HMaster.java | 18 +-
.../hadoop/hbase/master/MasterRpcServices.java | 7 +-
.../assignment/MergeTableRegionsProcedure.java | 10 +-
.../assignment/SplitTableRegionProcedure.java | 6 +-
.../hadoop/hbase/master/cleaner/CleanerChore.java | 2 +-
.../apache/hadoop/hbase/regionserver/HRegion.java | 2 +-
.../hadoop/hbase/regionserver/HRegionServer.java | 622 +++++++++------------
.../apache/hadoop/hbase/regionserver/HStore.java | 8 +-
.../hadoop/hbase/regionserver/HStoreFile.java | 141 +++--
.../{Leases.java => LeaseManager.java} | 44 +-
.../hadoop/hbase/regionserver/MemStoreFlusher.java | 4 +-
.../MetricsRegionServerWrapperImpl.java | 14 +-
.../MetricsTableWrapperAggregateImpl.java | 2 +-
.../hadoop/hbase/regionserver/RSDumpServlet.java | 7 +-
.../hadoop/hbase/regionserver/RSRpcServices.java | 187 ++++---
.../hbase/regionserver/RegionServerServices.java | 2 +-
.../hadoop/hbase/regionserver/SplitRequest.java | 11 +-
.../hbase/regionserver/SplitWALCallable.java | 2 +-
.../hadoop/hbase/regionserver/StoreFileInfo.java | 177 +++---
.../hadoop/hbase/regionserver/StoreFileReader.java | 62 +-
.../hbase/regionserver/compactions/Compactor.java | 4 +-
.../hadoop/hbase/tool/BulkLoadHFilesTool.java | 17 +-
.../apache/hadoop/hbase/util/CompressionTest.java | 1 -
.../org/apache/hadoop/hbase/util/HBaseFsck.java | 1 -
.../hadoop/hbase/util/ServerRegionReplicaUtil.java | 2 +-
.../main/resources/hbase-webapps/master/quotas.jsp | 33 +-
.../hadoop/hbase/HFilePerformanceEvaluation.java | 1 -
.../hadoop/hbase/MockRegionServerServices.java | 4 +-
.../apache/hadoop/hbase/TestCachedClusterId.java | 90 +++
.../hadoop/hbase/io/TestHalfStoreFileReader.java | 19 +-
.../apache/hadoop/hbase/io/hfile/TestChecksum.java | 46 +-
.../apache/hadoop/hbase/io/hfile/TestHFile.java | 35 +-
.../hadoop/hbase/io/hfile/TestHFileBlock.java | 50 +-
.../hadoop/hbase/io/hfile/TestHFileBlockIndex.java | 9 +-
.../hadoop/hbase/io/hfile/TestHFileEncryption.java | 10 +-
.../hadoop/hbase/io/hfile/TestHFileReaderImpl.java | 154 +++++
.../hfile/TestHFileScannerImplReferenceCount.java | 32 +-
.../hadoop/hbase/io/hfile/TestHFileSeek.java | 7 +-
.../hadoop/hbase/io/hfile/TestHFileWriterV3.java | 28 +-
.../io/hfile/TestLazyDataBlockDecompression.java | 13 +-
.../apache/hadoop/hbase/io/hfile/TestReseekTo.java | 1 -
.../apache/hadoop/hbase/io/hfile/TestSeekTo.java | 4 -
.../hadoop/hbase/master/MockRegionServer.java | 4 +-
.../hbase/master/TestMasterStatusServlet.java | 2 +-
.../TestCleanupCompactedFileAfterFailover.java | 2 +-
.../regionserver/TestEncryptionKeyRotation.java | 1 -
.../regionserver/TestEncryptionRandomKeying.java | 1 -
.../hadoop/hbase/regionserver/TestHStore.java | 2 +-
.../hadoop/hbase/regionserver/TestHStoreFile.java | 34 +-
.../regionserver/TestPerColumnFamilyFlush.java | 2 +-
.../hadoop/hbase/regionserver/TestPriorityRpc.java | 39 +-
.../hbase/regionserver/TestRSStatusServlet.java | 2 +-
.../regionserver/TestRegionServerAbortTimeout.java | 7 +-
.../regionserver/TestRegionServerMetrics.java | 4 +-
.../regionserver/TestRegionServerNoMaster.java | 2 +-
.../regionserver/TestRowPrefixBloomFilter.java | 18 +-
.../regionserver/TestSecureBulkLoadManager.java | 4 +-
.../hbase/regionserver/TestSplitWalDataLoss.java | 6 +-
.../hbase/regionserver/TestStoreFileInfo.java | 21 +-
.../TestStoreFileScannerWithTagCompression.java | 8 +-
.../hbase/regionserver/TestSwitchToStreamRead.java | 13 +-
.../replication/SerialReplicationTestBase.java | 7 +-
.../regionserver/TestReplicationSourceManager.java | 6 +-
.../hadoop/hbase/tool/TestBulkLoadHFiles.java | 1 -
.../hadoop/hbase/util/TestHBaseFsckEncryption.java | 1 -
src/main/asciidoc/_chapters/compression.adoc | 5 -
src/main/asciidoc/_chapters/cp.adoc | 33 --
src/main/asciidoc/_chapters/developer.adoc | 2 +-
.../asciidoc/_chapters/inmemory_compaction.adoc | 6 +-
152 files changed, 3643 insertions(+), 1839 deletions(-)
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/AddCPULoadAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CorruptDataFilesAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/CorruptPackagesCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/DelayPackagesCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/DeleteDataFilesAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/DuplicatePackagesCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/FillDiskCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/LosePackagesCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ReorderPackagesCommandAction.java
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/SudoCommandAction.java
copy hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/{ForceBalancerAction.java => TCCommandAction.java} (72%)
copy hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/{MasterKillingMonkeyFactory.java => DataIssuesMonkeyFactory.java} (56%)
create mode 100644 hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/DistributedIssuesMonkeyFactory.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePreadReader.java
copy hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/{InvalidHFileException.java => HFileStreamReader.java} (64%)
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ReaderContext.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ReaderContextBuilder.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/CachedClusterId.java
rename hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/{Leases.java => LeaseManager.java} (89%)
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderImpl.java
[hbase] 04/14: HBASE-22695 Store the rsgroup of a table in table
configuration (#426)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit ed2d5b9aba7ac347dbe260cc1c632cb66758ea15
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Sat Aug 3 07:53:27 2019 +0800
HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../org/apache/hadoop/hbase/HTableDescriptor.java | 6 +
.../hadoop/hbase/client/TableDescriptor.java | 8 +
.../hbase/client/TableDescriptorBuilder.java | 19 ++
.../apache/hadoop/hbase/rsgroup/RSGroupInfo.java | 42 ++-
.../org/apache/hadoop/hbase/master/HMaster.java | 4 +-
.../apache/hadoop/hbase/master/LoadBalancer.java | 49 +---
.../hbase/master/assignment/AssignmentManager.java | 6 +-
.../apache/hadoop/hbase/rsgroup/RSGroupAdmin.java | 23 --
.../hadoop/hbase/rsgroup/RSGroupAdminClient.java | 13 +-
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 152 ++++------
.../hadoop/hbase/rsgroup/RSGroupAdminServer.java | 311 ++++++---------------
.../hbase/rsgroup/RSGroupAdminServiceImpl.java | 111 ++++++--
.../hbase/rsgroup/RSGroupBasedLoadBalancer.java | 47 ++--
.../hadoop/hbase/rsgroup/RSGroupInfoManager.java | 23 --
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 115 +-------
.../apache/hadoop/hbase/rsgroup/RSGroupUtil.java | 113 ++++++++
.../hadoop/hbase/master/TestRegionPlacement2.java | 6 +-
.../balancer/RSGroupableBalancerTestBase.java | 84 +++---
.../balancer/TestRSGroupBasedLoadBalancer.java | 42 ++-
...lancerWithStochasticLoadBalancerAsInternal.java | 4 +-
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java | 1 -
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java | 104 +------
.../hadoop/hbase/rsgroup/TestRSGroupsBalance.java | 20 +-
.../hadoop/hbase/rsgroup/TestRSGroupsBase.java | 8 +-
.../hbase/rsgroup/TestRSGroupsOfflineMode.java | 6 +-
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 67 +++--
26 files changed, 590 insertions(+), 794 deletions(-)
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
index 8866eba..188bed6 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
@@ -23,6 +23,7 @@ import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@@ -987,4 +988,9 @@ public class HTableDescriptor implements TableDescriptor, Comparable<HTableDescr
protected ModifyableTableDescriptor getDelegateeForModification() {
return delegatee;
}
+
+ @Override
+ public Optional<String> getRegionServerGroup() {
+ return delegatee.getRegionServerGroup();
+ }
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java
index fc5e69e..a452387 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java
@@ -23,6 +23,7 @@ import java.util.Collection;
import java.util.Comparator;
import java.util.Iterator;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.stream.Stream;
import org.apache.hadoop.hbase.HConstants;
@@ -184,6 +185,13 @@ public interface TableDescriptor {
String getOwnerString();
/**
+ * Get the region server group this table belongs to. The regions of this table will be placed
+ * only on the region servers within this group. If not present, will be placed on
+ * {@link org.apache.hadoop.hbase.rsgroup.RSGroupInfo#DEFAULT_GROUP}.
+ */
+ Optional<String> getRegionServerGroup();
+
+ /**
* Getter for accessing the metadata associated with the key.
*
* @param key The key.
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
index 037a7f8..09ee0c5 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.hbase.Coprocessor;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.yetus.audience.InterfaceAudience;
@@ -188,6 +189,9 @@ public class TableDescriptorBuilder {
private static final Bytes PRIORITY_KEY
= new Bytes(Bytes.toBytes(PRIORITY));
+ private static final Bytes RSGROUP_KEY =
+ new Bytes(Bytes.toBytes(RSGroupInfo.TABLE_DESC_PROP_GROUP));
+
/**
* Relative priority of the table used for rpc scheduling
*/
@@ -537,6 +541,11 @@ public class TableDescriptorBuilder {
return this;
}
+ public TableDescriptorBuilder setRegionServerGroup(String group) {
+ desc.setValue(RSGROUP_KEY, new Bytes(Bytes.toBytes(group)));
+ return this;
+ }
+
public TableDescriptor build() {
return new ModifyableTableDescriptor(desc);
}
@@ -1577,6 +1586,16 @@ public class TableDescriptorBuilder {
public int getColumnFamilyCount() {
return families.size();
}
+
+ @Override
+ public Optional<String> getRegionServerGroup() {
+ Bytes value = values.get(RSGROUP_KEY);
+ if (value != null) {
+ return Optional.of(Bytes.toString(value.get(), value.getOffset(), value.getLength()));
+ } else {
+ return Optional.empty();
+ }
+ }
}
private static Optional<CoprocessorDescriptor> toCoprocessorDescriptor(String spec) {
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
index 25e827d..ad55d1f 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
@@ -34,21 +34,38 @@ import org.apache.yetus.audience.InterfaceAudience;
public class RSGroupInfo {
public static final String DEFAULT_GROUP = "default";
public static final String NAMESPACE_DESC_PROP_GROUP = "hbase.rsgroup.name";
+ public static final String TABLE_DESC_PROP_GROUP = "hbase.rsgroup.name";
private final String name;
// Keep servers in a sorted set so has an expected ordering when displayed.
private final SortedSet<Address> servers;
// Keep tables sorted too.
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
+ */
+ @Deprecated
private final SortedSet<TableName> tables;
public RSGroupInfo(String name) {
this(name, new TreeSet<Address>(), new TreeSet<TableName>());
}
+ RSGroupInfo(String name, SortedSet<Address> servers) {
+ this.name = name;
+ this.servers = servers == null ? new TreeSet<>() : new TreeSet<>(servers);
+ this.tables = new TreeSet<>();
+ }
+
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information for a table will be
+ * stored in the configuration of a table so this will be removed.
+ */
+ @Deprecated
RSGroupInfo(String name, SortedSet<Address> servers, SortedSet<TableName> tables) {
this.name = name;
this.servers = (servers == null) ? new TreeSet<>() : new TreeSet<>(servers);
- this.tables = (tables == null) ? new TreeSet<>() : new TreeSet<>(tables);
+ this.tables = (tables == null) ? new TreeSet<>() : new TreeSet<>(tables);
}
public RSGroupInfo(RSGroupInfo src) {
@@ -100,23 +117,46 @@ public class RSGroupInfo {
/**
* Get set of tables that are members of the group.
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
*/
+ @Deprecated
public SortedSet<TableName> getTables() {
return tables;
}
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
+ */
+ @Deprecated
public void addTable(TableName table) {
tables.add(table);
}
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
+ */
+ @Deprecated
public void addAllTables(Collection<TableName> arg) {
tables.addAll(arg);
}
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
+ */
+ @Deprecated
public boolean containsTable(TableName table) {
return tables.contains(table);
}
+ /**
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. The rsgroup information will be stored in
+ * the configuration of a table so this will be removed.
+ */
+ @Deprecated
public boolean removeTable(TableName table) {
return tables.remove(table);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 43da8c1..3a33e5f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -2022,7 +2022,7 @@ public class HMaster extends HRegionServer implements MasterServices {
// Replace with an async implementation from which you can get
// a success/failure result.
@VisibleForTesting
- public void move(final byte[] encodedRegionName, byte[] destServerName) throws HBaseIOException {
+ public void move(final byte[] encodedRegionName, byte[] destServerName) throws IOException {
RegionState regionState = assignmentManager.getRegionStates().
getRegionState(Bytes.toString(encodedRegionName));
@@ -3634,7 +3634,7 @@ public class HMaster extends HRegionServer implements MasterServices {
* @param servers Region servers to decommission.
*/
public void decommissionRegionServers(final List<ServerName> servers, final boolean offload)
- throws HBaseIOException {
+ throws IOException {
List<ServerName> serversAdded = new ArrayList<>(servers.size());
// Place the decommission marker first.
String parentZnode = getZooKeeper().getZNodePaths().drainingZNode;
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
index 816636f..0fc544a 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
@@ -19,12 +19,12 @@
package org.apache.hadoop.hbase.master;
import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.IOException;
import java.util.List;
import java.util.Map;
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.ClusterMetrics;
-import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.Stoppable;
import org.apache.hadoop.hbase.TableName;
@@ -65,95 +65,72 @@ public interface LoadBalancer extends Configurable, Stoppable, ConfigurationObse
ServerName BOGUS_SERVER_NAME = ServerName.valueOf("localhost,1,1");
/**
- * Set the current cluster status. This allows a LoadBalancer to map host name to a server
- * @param st
+ * Set the current cluster status. This allows a LoadBalancer to map host name to a server
*/
void setClusterMetrics(ClusterMetrics st);
/**
* Pass RegionStates and allow balancer to set the current cluster load.
- * @param ClusterLoad
*/
void setClusterLoad(Map<TableName, Map<ServerName, List<RegionInfo>>> ClusterLoad);
/**
* Set the master service.
- * @param masterServices
*/
void setMasterServices(MasterServices masterServices);
/**
* Perform the major balance operation
- * @param tableName
- * @param clusterState
* @return List of plans
*/
- List<RegionPlan> balanceCluster(TableName tableName, Map<ServerName,
- List<RegionInfo>> clusterState) throws HBaseIOException;
+ List<RegionPlan> balanceCluster(TableName tableName,
+ Map<ServerName, List<RegionInfo>> clusterState) throws IOException;
/**
* Perform the major balance operation
- * @param clusterState
* @return List of plans
*/
- List<RegionPlan> balanceCluster(Map<ServerName,
- List<RegionInfo>> clusterState) throws HBaseIOException;
+ List<RegionPlan> balanceCluster(Map<ServerName, List<RegionInfo>> clusterState)
+ throws IOException;
/**
* Perform a Round Robin assignment of regions.
- * @param regions
- * @param servers
* @return Map of servername to regioninfos
*/
- Map<ServerName, List<RegionInfo>> roundRobinAssignment(
- List<RegionInfo> regions,
- List<ServerName> servers
- ) throws HBaseIOException;
+ Map<ServerName, List<RegionInfo>> roundRobinAssignment(List<RegionInfo> regions,
+ List<ServerName> servers) throws IOException;
/**
* Assign regions to the previously hosting region server
- * @param regions
- * @param servers
* @return List of plans
*/
@Nullable
- Map<ServerName, List<RegionInfo>> retainAssignment(
- Map<RegionInfo, ServerName> regions,
- List<ServerName> servers
- ) throws HBaseIOException;
+ Map<ServerName, List<RegionInfo>> retainAssignment(Map<RegionInfo, ServerName> regions,
+ List<ServerName> servers) throws IOException;
/**
* Get a random region server from the list
* @param regionInfo Region for which this selection is being done.
- * @param servers
- * @return Servername
*/
- ServerName randomAssignment(
- RegionInfo regionInfo, List<ServerName> servers
- ) throws HBaseIOException;
+ ServerName randomAssignment(RegionInfo regionInfo, List<ServerName> servers) throws IOException;
/**
* Initialize the load balancer. Must be called after setters.
- * @throws HBaseIOException
*/
- void initialize() throws HBaseIOException;
+ void initialize() throws IOException;
/**
* Marks the region as online at balancer.
- * @param regionInfo
- * @param sn
*/
void regionOnline(RegionInfo regionInfo, ServerName sn);
/**
* Marks the region as offline at balancer.
- * @param regionInfo
*/
void regionOffline(RegionInfo regionInfo);
- /*
+ /**
* Notification that config has changed
- * @param conf
*/
@Override
void onConfigurationChange(Configuration conf);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
index c318540..debdf68 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
@@ -684,7 +684,7 @@ public class AssignmentManager {
this.master.getServerManager().createDestinationServersList(serversToExclude));
// Return mid-method!
return createAssignProcedures(assignments);
- } catch (HBaseIOException hioe) {
+ } catch (IOException hioe) {
LOG.warn("Failed roundRobinAssignment", hioe);
}
// If an error above, fall-through to this simpler assign. Last resort.
@@ -1999,7 +1999,7 @@ public class AssignmentManager {
}
try {
acceptPlan(regions, balancer.retainAssignment(retainMap, servers));
- } catch (HBaseIOException e) {
+ } catch (IOException e) {
LOG.warn("unable to retain assignment", e);
addToPendingAssignment(regions, retainMap.keySet());
}
@@ -2014,7 +2014,7 @@ public class AssignmentManager {
}
try {
acceptPlan(regions, balancer.roundRobinAssignment(hris, servers));
- } catch (HBaseIOException e) {
+ } catch (IOException e) {
LOG.warn("unable to round-robin assignment", e);
addToPendingAssignment(regions, hris);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
index 9ea996b..344d0b3 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
@@ -20,8 +20,6 @@ package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
import java.util.List;
import java.util.Set;
-
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -36,22 +34,11 @@ public interface RSGroupAdmin {
RSGroupInfo getRSGroupInfo(String groupName) throws IOException;
/**
- * Gets {@code RSGroupInfo} for the given table's group.
- */
- RSGroupInfo getRSGroupInfoOfTable(TableName tableName) throws IOException;
-
- /**
* Move given set of servers to the specified target RegionServer group.
*/
void moveServers(Set<Address> servers, String targetGroup) throws IOException;
/**
- * Move given set of tables to the specified target RegionServer group.
- * This will unassign all of a table's region so it can be reassigned to the correct group.
- */
- void moveTables(Set<TableName> tables, String targetGroup) throws IOException;
-
- /**
* Creates a new RegionServer group with the given name.
*/
void addRSGroup(String groupName) throws IOException;
@@ -80,16 +67,6 @@ public interface RSGroupAdmin {
RSGroupInfo getRSGroupOfServer(Address hostPort) throws IOException;
/**
- * Move given set of servers and tables to the specified target RegionServer group.
- * @param servers set of servers to move
- * @param tables set of tables to move
- * @param targetGroup the target group name
- * @throws IOException if moving the server and tables fail
- */
- void moveServersAndTables(Set<Address> servers, Set<TableName> tables,
- String targetGroup) throws IOException;
-
- /**
* Remove decommissioned servers from rsgroup.
* 1. Sometimes we may find the server aborted due to some hardware failure and we must offline
* the server for repairing. Or we need to move some servers to join other clusters.
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
index e7ab7f2..07f0efd 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServe
import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
/**
@@ -62,12 +63,17 @@ public class RSGroupAdminClient implements RSGroupAdmin {
stub = RSGroupAdminService.newBlockingStub(admin.coprocessorService());
}
+ // for writing UTs
+ @VisibleForTesting
+ protected RSGroupAdminClient() {
+ }
+
@Override
public RSGroupInfo getRSGroupInfo(String groupName) throws IOException {
try {
GetRSGroupInfoResponse resp = stub.getRSGroupInfo(null,
- GetRSGroupInfoRequest.newBuilder().setRSGroupName(groupName).build());
- if(resp.hasRSGroupInfo()) {
+ GetRSGroupInfoRequest.newBuilder().setRSGroupName(groupName).build());
+ if (resp.hasRSGroupInfo()) {
return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
return null;
@@ -76,7 +82,6 @@ public class RSGroupAdminClient implements RSGroupAdmin {
}
}
- @Override
public RSGroupInfo getRSGroupInfoOfTable(TableName tableName) throws IOException {
GetRSGroupInfoOfTableRequest request = GetRSGroupInfoOfTableRequest.newBuilder().setTableName(
ProtobufUtil.toProtoTableName(tableName)).build();
@@ -111,7 +116,6 @@ public class RSGroupAdminClient implements RSGroupAdmin {
}
}
- @Override
public void moveTables(Set<TableName> tables, String targetGroup) throws IOException {
MoveTablesRequest.Builder builder = MoveTablesRequest.newBuilder().setTargetGroup(targetGroup);
for(TableName tableName: tables) {
@@ -192,7 +196,6 @@ public class RSGroupAdminClient implements RSGroupAdmin {
}
}
- @Override
public void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String targetGroup)
throws IOException {
MoveServersAndTablesRequest.Builder builder =
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 2d5af04..3c4530f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -27,13 +27,10 @@ import java.util.stream.Collectors;
import org.apache.hadoop.hbase.CoprocessorEnvironment;
import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.HConstants;
-import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.NamespaceDescriptor;
-import org.apache.hadoop.hbase.PleaseHoldException;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
-import org.apache.hadoop.hbase.client.SnapshotDescription;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.coprocessor.CoreCoprocessor;
@@ -47,21 +44,16 @@ import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
-import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
// TODO: Encapsulate MasterObserver functions into separate subclass.
@CoreCoprocessor
@InterfaceAudience.Private
public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
- static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminEndpoint.class);
-
- private MasterServices master;
// Only instance of RSGroupInfoManager. RSGroup aware load balancers ask for this instance on
// their setup.
+ private MasterServices master;
private RSGroupInfoManager groupInfoManager;
private RSGroupAdminServer groupAdminServer;
private RSGroupAdminServiceImpl groupAdminService = new RSGroupAdminServiceImpl();
@@ -110,117 +102,91 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
return groupAdminService;
}
- private void assignTableToGroup(TableDescriptor desc) throws IOException {
- String groupName =
- master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
- .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (groupName == null) {
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo == null) {
- throw new ConstraintException(
- "Default RSGroup (" + groupName + ") for this table's namespace does not exist.");
- }
- if (!rsGroupInfo.containsTable(desc.getTableName())) {
- LOG.debug("Pre-moving table " + desc.getTableName() + " to RSGroup " + groupName);
- groupAdminServer.moveTables(Sets.newHashSet(desc.getTableName()), groupName);
- }
- }
-
/////////////////////////////////////////////////////////////////////////////
// MasterObserver overrides
/////////////////////////////////////////////////////////////////////////////
- private boolean rsgroupHasServersOnline(TableDescriptor desc) throws IOException {
- String groupName;
- try {
- groupName = master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
- .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (groupName == null) {
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
- } catch (MasterNotRunningException | PleaseHoldException e) {
- LOG.info("Master has not initialized yet; temporarily using default RSGroup '" +
- RSGroupInfo.DEFAULT_GROUP + "' for deploy of system table");
- groupName = RSGroupInfo.DEFAULT_GROUP;
+ @Override
+ public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
+ Set<Address> clearedServer =
+ servers.stream().filter(server -> !notClearedServers.contains(server))
+ .map(ServerName::getAddress).collect(Collectors.toSet());
+ if (!clearedServer.isEmpty()) {
+ groupAdminServer.removeServers(clearedServer);
}
+ }
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo == null) {
- throw new ConstraintException(
- "Default RSGroup (" + groupName + ") for this table's " + "namespace does not exist.");
+ private void checkGroupExists(Optional<String> optGroupName) throws IOException {
+ if (optGroupName.isPresent()) {
+ String groupName = optGroupName.get();
+ if (groupAdminServer.getRSGroupInfo(groupName) == null) {
+ throw new ConstraintException("Region server group " + groupName + " does not exit");
+ }
}
+ }
- for (ServerName onlineServer : master.getServerManager().createDestinationServersList()) {
- if (rsGroupInfo.getServers().contains(onlineServer.getAddress())) {
+ private boolean rsgroupHasServersOnline(TableDescriptor desc) throws IOException {
+ RSGroupInfo rsGroupInfo;
+ Optional<String> optGroupName = desc.getRegionServerGroup();
+ if (optGroupName.isPresent()) {
+ String groupName = optGroupName.get();
+ if (groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
+ // do not check for default group
+ return true;
+ }
+ rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
+ if (rsGroupInfo == null) {
+ throw new ConstraintException(
+ "RSGroup " + groupName + " for table " + desc.getTableName() + " does not exist");
+ }
+ } else {
+ NamespaceDescriptor nd =
+ master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString());
+ String groupNameOfNs = nd.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (groupNameOfNs == null || groupNameOfNs.equals(RSGroupInfo.DEFAULT_GROUP)) {
+ // do not check for default group
return true;
}
+ rsGroupInfo = groupAdminServer.getRSGroupInfo(groupNameOfNs);
+ if (rsGroupInfo == null) {
+ throw new ConstraintException("RSGroup " + groupNameOfNs + " for table " +
+ desc.getTableName() + "(inherit from namespace) does not exist");
+ }
}
- return false;
+ return master.getServerManager().createDestinationServersList().stream()
+ .anyMatch(onlineServer -> rsGroupInfo.containsServer(onlineServer.getAddress()));
}
@Override
- public void preCreateTableAction(final ObserverContext<MasterCoprocessorEnvironment> ctx,
- final TableDescriptor desc, final RegionInfo[] regions) throws IOException {
+ public void preCreateTableAction(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ TableDescriptor desc, RegionInfo[] regions) throws IOException {
+ checkGroupExists(desc.getRegionServerGroup());
if (!desc.getTableName().isSystemTable() && !rsgroupHasServersOnline(desc)) {
- throw new HBaseIOException("No online servers in the rsgroup, which table " +
- desc.getTableName().getNameAsString() + " belongs to");
+ throw new HBaseIOException("No online servers in the rsgroup for " + desc);
}
}
- // Assign table to default RSGroup.
- @Override
- public void postCreateTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableDescriptor desc, RegionInfo[] regions) throws IOException {
- assignTableToGroup(desc);
- }
-
- // Remove table from its RSGroup.
@Override
- public void postDeleteTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableName tableName) throws IOException {
- try {
- RSGroupInfo group = groupAdminServer.getRSGroupInfoOfTable(tableName);
- if (group != null) {
- LOG.debug(String.format("Removing deleted table '%s' from rsgroup '%s'", tableName,
- group.getName()));
- groupAdminServer.moveTables(Sets.newHashSet(tableName), null);
- }
- } catch (IOException ex) {
- LOG.debug("Failed to perform RSGroup information cleanup for table: " + tableName, ex);
- }
+ public TableDescriptor preModifyTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ TableName tableName, TableDescriptor currentDescriptor, TableDescriptor newDescriptor)
+ throws IOException {
+ checkGroupExists(newDescriptor.getRegionServerGroup());
+ return MasterObserver.super.preModifyTable(ctx, tableName, currentDescriptor, newDescriptor);
}
@Override
public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
NamespaceDescriptor ns) throws IOException {
- String group = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (group != null && groupAdminServer.getRSGroupInfo(group) == null) {
- throw new ConstraintException("Region server group " + group + " does not exit");
- }
+ checkGroupExists(
+ Optional.ofNullable(ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP)));
}
@Override
public void preModifyNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor currentNsDesc, NamespaceDescriptor newNsDesc) throws IOException {
- preCreateNamespace(ctx, newNsDesc);
- }
-
- @Override
- public void preCloneSnapshot(ObserverContext<MasterCoprocessorEnvironment> ctx,
- SnapshotDescription snapshot, TableDescriptor desc) throws IOException {
- assignTableToGroup(desc);
- }
-
- @Override
- public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
- List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
- Set<Address> clearedServer =
- servers.stream().filter(server -> !notClearedServers.contains(server))
- .map(ServerName::getAddress).collect(Collectors.toSet());
- if (!clearedServer.isEmpty()) {
- groupAdminServer.removeServers(clearedServer);
- }
+ NamespaceDescriptor currentNsDescriptor, NamespaceDescriptor newNsDescriptor)
+ throws IOException {
+ checkGroupExists(Optional
+ .ofNullable(newNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP)));
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
index f3ef4fb..59950e1 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
@@ -26,15 +26,16 @@ import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.function.Function;
-
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.master.HMaster;
import org.apache.hadoop.hbase.master.LoadBalancer;
@@ -42,7 +43,6 @@ import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.master.RegionPlan;
import org.apache.hadoop.hbase.master.RegionState;
import org.apache.hadoop.hbase.master.ServerManager;
-import org.apache.hadoop.hbase.master.assignment.AssignmentManager;
import org.apache.hadoop.hbase.master.assignment.RegionStateNode;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -85,14 +85,6 @@ public class RSGroupAdminServer implements RSGroupAdmin {
return rsGroupInfoManager.getRSGroup(groupName);
}
- @Override
- public RSGroupInfo getRSGroupInfoOfTable(TableName tableName) throws IOException {
- // We are reading across two Maps in the below with out synchronizing across
- // them; should be safe most of the time.
- String groupName = rsGroupInfoManager.getRSGroupOfTable(tableName);
- return groupName == null? null: rsGroupInfoManager.getRSGroup(groupName);
- }
-
private void checkOnlineServersOnly(Set<Address> servers) throws ConstraintException {
// This uglyness is because we only have Address, not ServerName.
// Online servers are keyed by ServerName.
@@ -159,104 +151,24 @@ public class RSGroupAdminServer implements RSGroupAdmin {
}
/**
- * Check servers and tables.
- *
- * @param servers servers to move
- * @param tables tables to move
- * @param targetGroupName target group name
- * @throws IOException if nulls or if servers and tables not belong to the same group
- */
- private void checkServersAndTables(Set<Address> servers, Set<TableName> tables,
- String targetGroupName) throws IOException {
- // Presume first server's source group. Later ensure all servers are from this group.
- Address firstServer = servers.iterator().next();
- RSGroupInfo tmpSrcGrp = rsGroupInfoManager.getRSGroupOfServer(firstServer);
- if (tmpSrcGrp == null) {
- // Be careful. This exception message is tested for in TestRSGroupsBase...
- throw new ConstraintException("Source RSGroup for server " + firstServer
- + " does not exist.");
- }
- RSGroupInfo srcGrp = new RSGroupInfo(tmpSrcGrp);
-
- // Only move online servers
- checkOnlineServersOnly(servers);
-
- // Ensure all servers are of same rsgroup.
- for (Address server: servers) {
- String tmpGroup = rsGroupInfoManager.getRSGroupOfServer(server).getName();
- if (!tmpGroup.equals(srcGrp.getName())) {
- throw new ConstraintException("Move server request should only come from one source " +
- "RSGroup. Expecting only " + srcGrp.getName() + " but contains " + tmpGroup);
- }
- }
-
- // Ensure all tables and servers are of same rsgroup.
- for (TableName table : tables) {
- String tmpGroup = rsGroupInfoManager.getRSGroupOfTable(table);
- if (!tmpGroup.equals(srcGrp.getName())) {
- throw new ConstraintException("Move table request should only come from one source " +
- "RSGroup. Expecting only " + srcGrp.getName() + " but contains " + tmpGroup);
- }
- }
-
- if (srcGrp.getServers().size() <= servers.size() && srcGrp.getTables().size() > tables.size()) {
- throw new ConstraintException("Cannot leave a RSGroup " + srcGrp.getName() +
- " that contains tables without servers to host them.");
- }
- }
-
- /**
- * Move every region from servers which are currently located on these servers,
- * but should not be located there.
- *
+ * Move every region from servers which are currently located on these servers, but should not be
+ * located there.
* @param servers the servers that will move to new group
* @param targetGroupName the target group name
* @throws IOException if moving the server and tables fail
*/
private void moveServerRegionsFromGroup(Set<Address> servers, String targetGroupName)
throws IOException {
- moveRegionsBetweenGroups(servers, targetGroupName,
- rs -> getRegions(rs),
- info -> {
- try {
- RSGroupInfo group = getRSGroupInfo(targetGroupName);
- return group.containsTable(info.getTable());
- } catch (IOException e) {
- e.printStackTrace();
- return false;
- }
- },
- rs -> rs.getHostname());
- }
-
- /**
- * Moves regions of tables which are not on target group servers.
- *
- * @param tables the tables that will move to new group
- * @param targetGroupName the target group name
- * @throws IOException if moving the region fails
- */
- private void moveTableRegionsToGroup(Set<TableName> tables, String targetGroupName)
- throws IOException {
- moveRegionsBetweenGroups(tables, targetGroupName,
- table -> {
- if (master.getAssignmentManager().isTableDisabled(table)) {
- return new ArrayList<>();
- }
- return master.getAssignmentManager().getRegionStates().getRegionsOfTable(table);
- },
- info -> {
- try {
- RSGroupInfo group = getRSGroupInfo(targetGroupName);
- ServerName sn =
- master.getAssignmentManager().getRegionStates().getRegionServerOfRegion(info);
- return group.containsServer(sn.getAddress());
- } catch (IOException e) {
- e.printStackTrace();
- return false;
- }
- },
- table -> table.getNameWithNamespaceInclAsString());
+ moveRegionsBetweenGroups(servers, targetGroupName, rs -> getRegions(rs), info -> {
+ try {
+ String groupName = RSGroupUtil.getRSGroupInfo(master, rsGroupInfoManager, info.getTable())
+ .map(RSGroupInfo::getName).orElse(RSGroupInfo.DEFAULT_GROUP);
+ return groupName.equals(targetGroupName);
+ } catch (IOException e) {
+ LOG.warn("Failed to test group for region {} and target group {}", info, targetGroupName);
+ return false;
+ }
+ }, rs -> rs.getHostname());
}
private <T> void moveRegionsBetweenGroups(Set<T> regionsOwners, String targetGroupName,
@@ -321,9 +233,6 @@ public class RSGroupAdminServer implements RSGroupAdmin {
}
}
- @edu.umd.cs.findbugs.annotations.SuppressWarnings(
- value="RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE",
- justification="Ignoring complaint because don't know what it is complaining about")
@Override
public void moveServers(Set<Address> servers, String targetGroupName) throws IOException {
if (servers == null) {
@@ -364,9 +273,16 @@ public class RSGroupAdminServer implements RSGroupAdmin {
"RSGroup. Expecting only " + srcGrp.getName() + " but contains " + tmpGroup);
}
}
- if (srcGrp.getServers().size() <= servers.size() && srcGrp.getTables().size() > 0) {
- throw new ConstraintException("Cannot leave a RSGroup " + srcGrp.getName() +
- " that contains tables without servers to host them.");
+ if (srcGrp.getServers().size() <= servers.size()) {
+ // check if there are still tables reference this group
+ for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
+ Optional<String> optGroupName = td.getRegionServerGroup();
+ if (optGroupName.isPresent() && optGroupName.get().equals(srcGrp.getName())) {
+ throw new ConstraintException(
+ "Cannot leave a RSGroup " + srcGrp.getName() + " that contains tables('" +
+ td.getTableName() + "' at least) without servers to host them.");
+ }
+ }
}
// MovedServers may be < passed in 'servers'.
@@ -378,38 +294,6 @@ public class RSGroupAdminServer implements RSGroupAdmin {
}
@Override
- public void moveTables(Set<TableName> tables, String targetGroup) throws IOException {
- if (tables == null) {
- throw new ConstraintException("The list of servers cannot be null.");
- }
- if (tables.size() < 1) {
- LOG.debug("moveTables() passed an empty set. Ignoring.");
- return;
- }
-
- // Hold a lock on the manager instance while moving servers to prevent
- // another writer changing our state while we are working.
- synchronized (rsGroupInfoManager) {
- if(targetGroup != null) {
- RSGroupInfo destGroup = rsGroupInfoManager.getRSGroup(targetGroup);
- if(destGroup == null) {
- throw new ConstraintException("Target " + targetGroup + " RSGroup does not exist.");
- }
- if(destGroup.getServers().size() < 1) {
- throw new ConstraintException("Target RSGroup must have at least one server.");
- }
- }
- rsGroupInfoManager.moveTables(tables, targetGroup);
-
- // targetGroup is null when a table is being deleted. In this case no further
- // action is required.
- if (targetGroup != null) {
- moveTableRegionsToGroup(tables, targetGroup);
- }
- }
- }
-
- @Override
public void addRSGroup(String name) throws IOException {
rsGroupInfoManager.addRSGroup(new RSGroupInfo(name));
}
@@ -423,17 +307,18 @@ public class RSGroupAdminServer implements RSGroupAdmin {
if (rsGroupInfo == null) {
throw new ConstraintException("RSGroup " + name + " does not exist");
}
- int tableCount = rsGroupInfo.getTables().size();
- if (tableCount > 0) {
- throw new ConstraintException("RSGroup " + name + " has " + tableCount +
- " tables; you must remove these tables from the rsgroup before " +
- "the rsgroup can be removed.");
- }
int serverCount = rsGroupInfo.getServers().size();
if (serverCount > 0) {
throw new ConstraintException("RSGroup " + name + " has " + serverCount +
- " servers; you must remove these servers from the RSGroup before" +
- "the RSGroup can be removed.");
+ " servers; you must remove these servers from the RSGroup before" +
+ " the RSGroup can be removed.");
+ }
+ for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
+ if (td.getRegionServerGroup().map(name::equals).orElse(false)) {
+ throw new ConstraintException("RSGroup " + name + " is already referenced by " +
+ td.getTableName() + "; you must remove all the tables from the rsgroup before " +
+ "the rsgroup can be removed.");
+ }
}
for (NamespaceDescriptor ns : master.getClusterSchema().getNamespaces()) {
String nsGroup = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
@@ -458,27 +343,29 @@ public class RSGroupAdminServer implements RSGroupAdmin {
}
if (getRSGroupInfo(groupName) == null) {
- throw new ConstraintException("RSGroup does not exist: "+groupName);
+ throw new ConstraintException("RSGroup does not exist: " + groupName);
}
// Only allow one balance run at at time.
Map<String, RegionState> groupRIT = rsGroupGetRegionsInTransition(groupName);
if (groupRIT.size() > 0) {
LOG.debug("Not running balancer because {} region(s) in transition: {}", groupRIT.size(),
- StringUtils.abbreviate(
- master.getAssignmentManager().getRegionStates().getRegionsInTransition().toString(),
- 256));
+ StringUtils.abbreviate(
+ master.getAssignmentManager().getRegionStates().getRegionsInTransition().toString(),
+ 256));
return false;
}
if (serverManager.areDeadServersInProgress()) {
LOG.debug("Not running balancer because processing dead regionserver(s): {}",
- serverManager.getDeadServers());
+ serverManager.getDeadServers());
return false;
}
- //We balance per group instead of per table
+ // We balance per group instead of per table
List<RegionPlan> plans = new ArrayList<>();
- for(Map.Entry<TableName, Map<ServerName, List<RegionInfo>>> tableMap:
- getRSGroupAssignmentsByTable(groupName).entrySet()) {
+ Map<TableName, Map<ServerName, List<RegionInfo>>> assignmentsByTable =
+ getRSGroupAssignmentsByTable(groupName);
+ for (Map.Entry<TableName, Map<ServerName, List<RegionInfo>>> tableMap : assignmentsByTable
+ .entrySet()) {
LOG.info("Creating partial plan for table {} : {}", tableMap.getKey(), tableMap.getValue());
List<RegionPlan> partialPlans = balancer.balanceCluster(tableMap.getValue());
LOG.info("Partial plan for table {} : {}", tableMap.getKey(), partialPlans);
@@ -507,100 +394,66 @@ public class RSGroupAdminServer implements RSGroupAdmin {
}
@Override
- public void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String targetGroup)
- throws IOException {
+ public void removeServers(Set<Address> servers) throws IOException {
if (servers == null || servers.isEmpty()) {
- throw new ConstraintException("The list of servers to move cannot be null or empty.");
- }
- if (tables == null || tables.isEmpty()) {
- throw new ConstraintException("The list of tables to move cannot be null or empty.");
+ throw new ConstraintException("The set of servers to remove cannot be null or empty.");
}
-
- //check target group
- getAndCheckRSGroupInfo(targetGroup);
-
- // Hold a lock on the manager instance while moving servers and tables to prevent
+ // Hold a lock on the manager instance while moving servers to prevent
// another writer changing our state while we are working.
synchronized (rsGroupInfoManager) {
- //check servers and tables status
- checkServersAndTables(servers, tables, targetGroup);
-
- //Move servers and tables to a new group.
- String srcGroup = getRSGroupOfServer(servers.iterator().next()).getName();
- rsGroupInfoManager.moveServersAndTables(servers, tables, srcGroup, targetGroup);
-
- //move regions on these servers which do not belong to group tables
- moveServerRegionsFromGroup(servers, targetGroup);
- //move regions of these tables which are not on group servers
- moveTableRegionsToGroup(tables, targetGroup);
+ // check the set of servers
+ checkForDeadOrOnlineServers(servers);
+ rsGroupInfoManager.removeServers(servers);
+ LOG.info("Remove decommissioned servers {} from RSGroup done", servers);
}
- LOG.info("Move servers and tables done. Severs: {}, Tables: {} => {}", servers, tables,
- targetGroup);
}
- @Override
- public void removeServers(Set<Address> servers) throws IOException {
- {
- if (servers == null || servers.isEmpty()) {
- throw new ConstraintException("The set of servers to remove cannot be null or empty.");
- }
- // Hold a lock on the manager instance while moving servers to prevent
- // another writer changing our state while we are working.
- synchronized (rsGroupInfoManager) {
- //check the set of servers
- checkForDeadOrOnlineServers(servers);
- rsGroupInfoManager.removeServers(servers);
- LOG.info("Remove decommissioned servers {} from RSGroup done", servers);
- }
+ private boolean isTableInGroup(TableName tableName, String groupName,
+ Set<TableName> tablesInGroupCache) throws IOException {
+ if (tablesInGroupCache.contains(tableName)) {
+ return true;
}
+ if (RSGroupUtil.getRSGroupInfo(master, rsGroupInfoManager, tableName).map(RSGroupInfo::getName)
+ .orElse(RSGroupInfo.DEFAULT_GROUP).equals(groupName)) {
+ tablesInGroupCache.add(tableName);
+ return true;
+ }
+ return false;
}
private Map<String, RegionState> rsGroupGetRegionsInTransition(String groupName)
- throws IOException {
+ throws IOException {
Map<String, RegionState> rit = Maps.newTreeMap();
- AssignmentManager am = master.getAssignmentManager();
- for(TableName tableName : getRSGroupInfo(groupName).getTables()) {
- for(RegionInfo regionInfo: am.getRegionStates().getRegionsOfTable(tableName)) {
- RegionState state = am.getRegionStates().getRegionTransitionState(regionInfo);
- if(state != null) {
- rit.put(regionInfo.getEncodedName(), state);
- }
+ Set<TableName> tablesInGroupCache = new HashSet<>();
+ for (RegionStateNode regionNode : master.getAssignmentManager().getRegionsInTransition()) {
+ TableName tn = regionNode.getTable();
+ if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
+ rit.put(regionNode.getRegionInfo().getEncodedName(), regionNode.toRegionState());
}
}
return rit;
}
private Map<TableName, Map<ServerName, List<RegionInfo>>>
- getRSGroupAssignmentsByTable(String groupName) throws IOException {
+ getRSGroupAssignmentsByTable(String groupName) throws IOException {
Map<TableName, Map<ServerName, List<RegionInfo>>> result = Maps.newHashMap();
- RSGroupInfo rsGroupInfo = getRSGroupInfo(groupName);
- Map<TableName, Map<ServerName, List<RegionInfo>>> assignments = Maps.newHashMap();
- for(Map.Entry<RegionInfo, ServerName> entry:
- master.getAssignmentManager().getRegionStates().getRegionAssignments().entrySet()) {
- TableName currTable = entry.getKey().getTable();
- ServerName currServer = entry.getValue();
- RegionInfo currRegion = entry.getKey();
- if (rsGroupInfo.getTables().contains(currTable)) {
- assignments.putIfAbsent(currTable, new HashMap<>());
- assignments.get(currTable).putIfAbsent(currServer, new ArrayList<>());
- assignments.get(currTable).get(currServer).add(currRegion);
+ Set<TableName> tablesInGroupCache = new HashSet<>();
+ for (Map.Entry<RegionInfo, ServerName> entry : master.getAssignmentManager().getRegionStates()
+ .getRegionAssignments().entrySet()) {
+ RegionInfo region = entry.getKey();
+ TableName tn = region.getTable();
+ ServerName server = entry.getValue();
+ if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
+ result.computeIfAbsent(tn, k -> new HashMap<>())
+ .computeIfAbsent(server, k -> new ArrayList<>()).add(region);
}
}
-
- Map<ServerName, List<RegionInfo>> serverMap = Maps.newHashMap();
- for(ServerName serverName: master.getServerManager().getOnlineServers().keySet()) {
- if(rsGroupInfo.getServers().contains(serverName.getAddress())) {
- serverMap.put(serverName, Collections.emptyList());
- }
- }
-
- // add all tables that are members of the group
- for(TableName tableName : rsGroupInfo.getTables()) {
- if(assignments.containsKey(tableName)) {
- result.put(tableName, new HashMap<>());
- result.get(tableName).putAll(serverMap);
- result.get(tableName).putAll(assignments.get(tableName));
- LOG.debug("Adding assignments for {}: {}", tableName, assignments.get(tableName));
+ RSGroupInfo rsGroupInfo = getRSGroupInfo(groupName);
+ for (ServerName serverName : master.getServerManager().getOnlineServers().keySet()) {
+ if (rsGroupInfo.containsServer(serverName.getAddress())) {
+ for (Map<ServerName, List<RegionInfo>> map : result.values()) {
+ map.computeIfAbsent(serverName, k -> Collections.emptyList());
+ }
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
index 918a4fe..6bc4519 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
@@ -20,14 +20,24 @@ package org.apache.hadoop.hbase.rsgroup;
import com.google.protobuf.RpcCallback;
import com.google.protobuf.RpcController;
import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
import org.apache.hadoop.hbase.ipc.RpcServer;
import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait;
import org.apache.hadoop.hbase.net.Address;
+import org.apache.hadoop.hbase.procedure2.Procedure;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos;
@@ -57,6 +67,8 @@ import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.hadoop.hbase.security.access.Permission.Action;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
@@ -68,6 +80,8 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
*/
class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
+ private static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminServiceImpl.class);
+
private MasterServices master;
private RSGroupAdminServer groupAdminServer;
@@ -107,12 +121,17 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
return userProvider.getCurrent();
}
+ // for backward compatible
+ private RSGroupInfo fillTables(RSGroupInfo rsGroupInfo) throws IOException {
+ return RSGroupUtil.fillTables(rsGroupInfo, master.getTableDescriptors().getAll().values());
+ }
+
@Override
public void getRSGroupInfo(RpcController controller, GetRSGroupInfoRequest request,
RpcCallback<GetRSGroupInfoResponse> done) {
GetRSGroupInfoResponse.Builder builder = GetRSGroupInfoResponse.newBuilder();
String groupName = request.getRSGroupName();
- RSGroupAdminEndpoint.LOG.info(
+ LOG.info(
master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, group=" + groupName);
try {
if (master.getMasterCoprocessorHost() != null) {
@@ -121,7 +140,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
checkPermission("getRSGroupInfo");
RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
if (rsGroupInfo != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(rsGroupInfo)));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfo(groupName);
@@ -137,17 +156,24 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
RpcCallback<GetRSGroupInfoOfTableResponse> done) {
GetRSGroupInfoOfTableResponse.Builder builder = GetRSGroupInfoOfTableResponse.newBuilder();
TableName tableName = ProtobufUtil.toTableName(request.getTableName());
- RSGroupAdminEndpoint.LOG.info(
+ LOG.info(
master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, table=" + tableName);
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preGetRSGroupInfoOfTable(tableName);
}
checkPermission("getRSGroupInfoOfTable");
- RSGroupInfo RSGroupInfo = groupAdminServer.getRSGroupInfoOfTable(tableName);
- if (RSGroupInfo != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ Optional<RSGroupInfo> optGroup =
+ RSGroupUtil.getRSGroupInfo(master, groupAdminServer, tableName);
+ if (optGroup.isPresent()) {
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(optGroup.get())));
+ } else {
+ if (master.getTableStateManager().isTablePresent(tableName)) {
+ RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(rsGroupInfo)));
+ }
}
+
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfoOfTable(tableName);
}
@@ -165,8 +191,8 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
for (HBaseProtos.ServerName el : request.getServersList()) {
hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
}
- RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +
- " to rsgroup " + request.getTargetGroup());
+ LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts + " to rsgroup " +
+ request.getTargetGroup());
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preMoveServers(hostPorts, request.getTargetGroup());
@@ -182,6 +208,27 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
done.run(builder.build());
}
+ private void moveTablesAndWait(Set<TableName> tables, String targetGroup) throws IOException {
+ List<Long> procIds = new ArrayList<Long>();
+ for (TableName tableName : tables) {
+ TableDescriptor oldTd = master.getTableDescriptors().get(tableName);
+ if (oldTd == null) {
+ continue;
+ }
+ TableDescriptor newTd =
+ TableDescriptorBuilder.newBuilder(oldTd).setRegionServerGroup(targetGroup).build();
+ procIds.add(master.modifyTable(tableName, newTd, HConstants.NO_NONCE, HConstants.NO_NONCE));
+ }
+ for (long procId : procIds) {
+ Procedure<?> proc = master.getMasterProcedureExecutor().getProcedure(procId);
+ if (proc == null) {
+ continue;
+ }
+ ProcedureSyncWait.waitForProcedureToCompleteIOE(master.getMasterProcedureExecutor(), proc,
+ Long.MAX_VALUE);
+ }
+ }
+
@Override
public void moveTables(RpcController controller, MoveTablesRequest request,
RpcCallback<MoveTablesResponse> done) {
@@ -190,14 +237,14 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
for (HBaseProtos.TableName tableName : request.getTableNameList()) {
tables.add(ProtobufUtil.toTableName(tableName));
}
- RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move tables " + tables +
- " to rsgroup " + request.getTargetGroup());
+ LOG.info(master.getClientIdAuditPrefix() + " move tables " + tables + " to rsgroup " +
+ request.getTargetGroup());
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preMoveTables(tables, request.getTargetGroup());
}
checkPermission("moveTables");
- groupAdminServer.moveTables(tables, request.getTargetGroup());
+ moveTablesAndWait(tables, request.getTargetGroup());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postMoveTables(tables, request.getTargetGroup());
}
@@ -211,8 +258,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
public void addRSGroup(RpcController controller, AddRSGroupRequest request,
RpcCallback<AddRSGroupResponse> done) {
AddRSGroupResponse.Builder builder = AddRSGroupResponse.newBuilder();
- RSGroupAdminEndpoint.LOG
- .info(master.getClientIdAuditPrefix() + " add rsgroup " + request.getRSGroupName());
+ LOG.info(master.getClientIdAuditPrefix() + " add rsgroup " + request.getRSGroupName());
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
@@ -232,8 +278,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
public void removeRSGroup(RpcController controller, RemoveRSGroupRequest request,
RpcCallback<RemoveRSGroupResponse> done) {
RemoveRSGroupResponse.Builder builder = RemoveRSGroupResponse.newBuilder();
- RSGroupAdminEndpoint.LOG
- .info(master.getClientIdAuditPrefix() + " remove rsgroup " + request.getRSGroupName());
+ LOG.info(master.getClientIdAuditPrefix() + " remove rsgroup " + request.getRSGroupName());
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preRemoveRSGroup(request.getRSGroupName());
@@ -253,7 +298,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
public void balanceRSGroup(RpcController controller, BalanceRSGroupRequest request,
RpcCallback<BalanceRSGroupResponse> done) {
BalanceRSGroupResponse.Builder builder = BalanceRSGroupResponse.newBuilder();
- RSGroupAdminEndpoint.LOG.info(
+ LOG.info(
master.getClientIdAuditPrefix() + " balance rsgroup, group=" + request.getRSGroupName());
try {
if (master.getMasterCoprocessorHost() != null) {
@@ -276,14 +321,28 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
public void listRSGroupInfos(RpcController controller, ListRSGroupInfosRequest request,
RpcCallback<ListRSGroupInfosResponse> done) {
ListRSGroupInfosResponse.Builder builder = ListRSGroupInfosResponse.newBuilder();
- RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " list rsgroup");
+ LOG.info(master.getClientIdAuditPrefix() + " list rsgroup");
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preListRSGroups();
}
checkPermission("listRSGroup");
- for (RSGroupInfo RSGroupInfo : groupAdminServer.listRSGroups()) {
- builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ List<RSGroupInfo> rsGroupInfos = groupAdminServer.listRSGroups().stream()
+ .map(RSGroupInfo::new).collect(Collectors.toList());
+ Map<String, RSGroupInfo> name2Info = new HashMap<>();
+ for (RSGroupInfo rsGroupInfo : rsGroupInfos) {
+ name2Info.put(rsGroupInfo.getName(), rsGroupInfo);
+ }
+ for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
+ String groupName = td.getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo rsGroupInfo = name2Info.get(groupName);
+ if (rsGroupInfo != null) {
+ rsGroupInfo.addTable(td.getTableName());
+ }
+ }
+ for (RSGroupInfo rsGroupInfo : rsGroupInfos) {
+ // TODO: this can be done at once outside this loop, do not need to scan all every time.
+ builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postListRSGroups();
@@ -300,8 +359,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
GetRSGroupInfoOfServerResponse.Builder builder = GetRSGroupInfoOfServerResponse.newBuilder();
Address hp =
Address.fromParts(request.getServer().getHostName(), request.getServer().getPort());
- RSGroupAdminEndpoint.LOG
- .info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, server=" + hp);
+ LOG.info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, server=" + hp);
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preGetRSGroupInfoOfServer(hp);
@@ -309,7 +367,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
checkPermission("getRSGroupInfoOfServer");
RSGroupInfo info = groupAdminServer.getRSGroupOfServer(hp);
if (info != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(info));
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(info)));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfoOfServer(hp);
@@ -332,15 +390,16 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
for (HBaseProtos.TableName tableName : request.getTableNameList()) {
tables.add(ProtobufUtil.toTableName(tableName));
}
- RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +
- " and tables " + tables + " to rsgroup" + request.getTargetGroup());
+ LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts + " and tables " +
+ tables + " to rsgroup" + request.getTargetGroup());
try {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preMoveServersAndTables(hostPorts, tables,
request.getTargetGroup());
}
checkPermission("moveServersAndTables");
- groupAdminServer.moveServersAndTables(hostPorts, tables, request.getTargetGroup());
+ groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
+ moveTablesAndWait(tables, request.getTargetGroup());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postMoveServersAndTables(hostPorts, tables,
request.getTargetGroup());
@@ -359,7 +418,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
for (HBaseProtos.ServerName el : request.getServersList()) {
servers.add(Address.fromParts(el.getHostName(), el.getPort()));
}
- RSGroupAdminEndpoint.LOG.info(
+ LOG.info(
master.getClientIdAuditPrefix() + " remove decommissioned servers from rsgroup: " + servers);
try {
if (master.getMasterCoprocessorHost() != null) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index be76805..0f943d0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -27,7 +27,6 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.ClusterMetrics;
import org.apache.hadoop.hbase.HBaseIOException;
@@ -111,13 +110,13 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
@Override
public List<RegionPlan> balanceCluster(TableName tableName, Map<ServerName, List<RegionInfo>>
- clusterState) throws HBaseIOException {
+ clusterState) throws IOException {
return balanceCluster(clusterState);
}
@Override
public List<RegionPlan> balanceCluster(Map<ServerName, List<RegionInfo>> clusterState)
- throws HBaseIOException {
+ throws IOException {
if (!isOnline()) {
throw new ConstraintException(
RSGroupInfoManager.class.getSimpleName() + " is not online, unable to perform balance");
@@ -168,8 +167,8 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
}
@Override
- public Map<ServerName, List<RegionInfo>> roundRobinAssignment(List<RegionInfo> regions,
- List<ServerName> servers) throws HBaseIOException {
+ public Map<ServerName, List<RegionInfo>> roundRobinAssignment(
+ List<RegionInfo> regions, List<ServerName> servers) throws IOException {
Map<ServerName, List<RegionInfo>> assignments = Maps.newHashMap();
ListMultimap<String, RegionInfo> regionMap = ArrayListMultimap.create();
ListMultimap<String, ServerName> serverMap = ArrayListMultimap.create();
@@ -198,12 +197,11 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
try {
Map<ServerName, List<RegionInfo>> assignments = new TreeMap<>();
ListMultimap<String, RegionInfo> groupToRegion = ArrayListMultimap.create();
+ RSGroupInfo defaultInfo = rsGroupInfoManager.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
for (RegionInfo region : regions.keySet()) {
- String groupName = rsGroupInfoManager.getRSGroupOfTable(region.getTable());
- if (groupName == null) {
- LOG.debug("Group not found for table " + region.getTable() + ", using default");
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
+ String groupName =
+ RSGroupUtil.getRSGroupInfo(masterServices, rsGroupInfoManager, region.getTable())
+ .orElse(defaultInfo).getName();
groupToRegion.put(groupName, region);
}
for (String key : groupToRegion.keySet()) {
@@ -234,7 +232,7 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
@Override
public ServerName randomAssignment(RegionInfo region,
- List<ServerName> servers) throws HBaseIOException {
+ List<ServerName> servers) throws IOException {
ListMultimap<String,RegionInfo> regionMap = LinkedListMultimap.create();
ListMultimap<String,ServerName> serverMap = LinkedListMultimap.create();
generateGroupMaps(Lists.newArrayList(region), servers, regionMap, serverMap);
@@ -246,12 +244,11 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
ListMultimap<String, RegionInfo> regionMap, ListMultimap<String, ServerName> serverMap)
throws HBaseIOException {
try {
+ RSGroupInfo defaultInfo = rsGroupInfoManager.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
for (RegionInfo region : regions) {
- String groupName = rsGroupInfoManager.getRSGroupOfTable(region.getTable());
- if (groupName == null) {
- LOG.debug("Group not found for table " + region.getTable() + ", using default");
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
+ String groupName =
+ RSGroupUtil.getRSGroupInfo(masterServices, rsGroupInfoManager, region.getTable())
+ .orElse(defaultInfo).getName();
regionMap.put(groupName, region);
}
for (String groupKey : regionMap.keySet()) {
@@ -299,11 +296,11 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
}
private Pair<Map<ServerName, List<RegionInfo>>, List<RegionPlan>> correctAssignments(
- Map<ServerName, List<RegionInfo>> existingAssignments) throws HBaseIOException{
+ Map<ServerName, List<RegionInfo>> existingAssignments) throws IOException {
// To return
Map<ServerName, List<RegionInfo>> correctAssignments = new TreeMap<>();
List<RegionPlan> regionPlansForMisplacedRegions = new ArrayList<>();
-
+ RSGroupInfo defaultInfo = rsGroupInfoManager.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
for (Map.Entry<ServerName, List<RegionInfo>> assignments : existingAssignments.entrySet()){
ServerName currentHostServer = assignments.getKey();
correctAssignments.put(currentHostServer, new LinkedList<>());
@@ -311,15 +308,11 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
for (RegionInfo region : regions) {
RSGroupInfo targetRSGInfo = null;
try {
- String groupName = rsGroupInfoManager.getRSGroupOfTable(region.getTable());
- if (groupName == null) {
- LOG.debug("Group not found for table " + region.getTable() + ", using default");
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
- targetRSGInfo = rsGroupInfoManager.getRSGroup(groupName);
+ targetRSGInfo =
+ RSGroupUtil.getRSGroupInfo(masterServices, rsGroupInfoManager, region.getTable())
+ .orElse(defaultInfo);
} catch (IOException exp) {
- LOG.debug("RSGroup information null for region of table " + region.getTable(),
- exp);
+ LOG.debug("RSGroup information null for region of table " + region.getTable(), exp);
}
if (targetRSGInfo == null ||
!targetRSGInfo.containsServer(currentHostServer.getAddress())) { // region is mis-placed
@@ -336,7 +329,7 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
}
@Override
- public void initialize() throws HBaseIOException {
+ public void initialize() throws IOException {
try {
if (rsGroupInfoManager == null) {
List<RSGroupAdminEndpoint> cps =
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 70aeabf..28f7c1f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -20,7 +20,6 @@ package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
import java.util.List;
import java.util.Set;
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -64,18 +63,6 @@ public interface RSGroupInfoManager {
RSGroupInfo getRSGroup(String groupName) throws IOException;
/**
- * Get the group membership of a table
- */
- String getRSGroupOfTable(TableName tableName) throws IOException;
-
- /**
- * Set the group membership of a set of tables
- * @param tableNames set of tables to move
- * @param groupName name of group of tables to move to
- */
- void moveTables(Set<TableName> tableNames, String groupName) throws IOException;
-
- /**
* List the existing {@code RSGroupInfo}s.
*/
List<RSGroupInfo> listRSGroups() throws IOException;
@@ -92,16 +79,6 @@ public interface RSGroupInfoManager {
boolean isOnline();
/**
- * Move servers and tables to a new group.
- * @param servers list of servers, must be part of the same group
- * @param tables set of tables to move
- * @param srcGroup groupName being moved from
- * @param dstGroup groupName being moved to
- */
- void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String srcGroup,
- String dstGroup) throws IOException;
-
- /**
* Remove decommissioned servers from rsgroup
* @param servers set of servers to remove
*/
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index 8aa7520..37f3ce6 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -23,10 +23,8 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.LinkedList;
import java.util.List;
import java.util.Map;
-import java.util.NavigableSet;
import java.util.OptionalLong;
import java.util.Set;
import java.util.SortedSet;
@@ -143,7 +141,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// There two Maps are immutable and wholesale replaced on each modification
// so are safe to access concurrently. See class comment.
private volatile Map<String, RSGroupInfo> rsGroupMap = Collections.emptyMap();
- private volatile Map<TableName, String> tableMap = Collections.emptyMap();
private final MasterServices masterServices;
private final AsyncClusterConnection conn;
@@ -261,44 +258,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
@Override
- public String getRSGroupOfTable(TableName tableName) {
- return tableMap.get(tableName);
- }
-
- @Override
- public synchronized void moveTables(Set<TableName> tableNames, String groupName)
- throws IOException {
- // Check if rsGroupMap contains the destination rsgroup
- if (groupName != null && !rsGroupMap.containsKey(groupName)) {
- throw new DoNotRetryIOException("Group " + groupName + " does not exist");
- }
-
- // Make a copy of rsGroupMap to update
- Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
-
- // Remove tables from their original rsgroups
- // and update the copy of rsGroupMap
- for (TableName tableName : tableNames) {
- if (tableMap.containsKey(tableName)) {
- RSGroupInfo src = new RSGroupInfo(newGroupMap.get(tableMap.get(tableName)));
- src.removeTable(tableName);
- newGroupMap.put(src.getName(), src);
- }
- }
-
- // Add tables to the destination rsgroup
- // and update the copy of rsGroupMap
- if (groupName != null) {
- RSGroupInfo dstGroup = new RSGroupInfo(newGroupMap.get(groupName));
- dstGroup.addAllTables(tableNames);
- newGroupMap.put(dstGroup.getName(), dstGroup);
- }
-
- // Flush according to the updated copy of rsGroupMap
- flushConfig(newGroupMap);
- }
-
- @Override
public synchronized void removeRSGroup(String groupName) throws IOException {
if (!rsGroupMap.containsKey(groupName) || groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
throw new DoNotRetryIOException(
@@ -311,7 +270,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public List<RSGroupInfo> listRSGroups() {
- return Lists.newLinkedList(rsGroupMap.values());
+ return Lists.newArrayList(rsGroupMap.values());
}
@Override
@@ -320,31 +279,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
@Override
- public void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String srcGroup,
- String dstGroup) throws IOException {
- // get server's group
- RSGroupInfo srcGroupInfo = getRSGroupInfo(srcGroup);
- RSGroupInfo dstGroupInfo = getRSGroupInfo(dstGroup);
-
- // move servers
- for (Address el : servers) {
- srcGroupInfo.removeServer(el);
- dstGroupInfo.addServer(el);
- }
- // move tables
- for (TableName tableName : tables) {
- srcGroupInfo.removeTable(tableName);
- dstGroupInfo.addTable(tableName);
- }
-
- // flush changed groupinfo
- Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
- newGroupMap.put(srcGroupInfo.getName(), srcGroupInfo);
- newGroupMap.put(dstGroupInfo.getName(), dstGroupInfo);
- flushConfig(newGroupMap);
- }
-
- @Override
public synchronized void removeServers(Set<Address> servers) throws IOException {
Map<String, RSGroupInfo> rsGroupInfos = new HashMap<String, RSGroupInfo>();
for (Address el : servers) {
@@ -425,7 +359,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
* startup of the manager.
*/
private synchronized void refresh(boolean forceOnline) throws IOException {
- List<RSGroupInfo> groupList = new LinkedList<>();
+ List<RSGroupInfo> groupList = new ArrayList<>();
// Overwrite anything read from zk, group table is source of truth
// if online read from GROUP table
@@ -437,37 +371,20 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
groupList.addAll(retrieveGroupListFromZookeeper());
}
- // refresh default group, prune
- NavigableSet<TableName> orphanTables = new TreeSet<>();
- for (String entry : masterServices.getTableDescriptors().getAll().keySet()) {
- orphanTables.add(TableName.valueOf(entry));
- }
- for (RSGroupInfo group : groupList) {
- if (!group.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
- orphanTables.removeAll(group.getTables());
- }
- }
-
// This is added to the last of the list so it overwrites the 'default' rsgroup loaded
// from region group table or zk
- groupList.add(new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, getDefaultServers(), orphanTables));
+ groupList.add(new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, getDefaultServers()));
// populate the data
HashMap<String, RSGroupInfo> newGroupMap = Maps.newHashMap();
- HashMap<TableName, String> newTableMap = Maps.newHashMap();
for (RSGroupInfo group : groupList) {
newGroupMap.put(group.getName(), group);
- for (TableName table : group.getTables()) {
- newTableMap.put(table, group.getName());
- }
}
- resetRSGroupAndTableMaps(newGroupMap, newTableMap);
+ resetRSGroupMap(newGroupMap);
updateCacheOfRSGroups(rsGroupMap.keySet());
}
- private synchronized Map<TableName, String> flushConfigTable(Map<String, RSGroupInfo> groupMap)
- throws IOException {
- Map<TableName, String> newTableMap = Maps.newHashMap();
+ private void flushConfigTable(Map<String, RSGroupInfo> groupMap) throws IOException {
List<Mutation> mutations = Lists.newArrayList();
// populate deletes
@@ -484,15 +401,11 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
Put p = new Put(Bytes.toBytes(RSGroupInfo.getName()));
p.addColumn(META_FAMILY_BYTES, META_QUALIFIER_BYTES, proto.toByteArray());
mutations.add(p);
- for (TableName entry : RSGroupInfo.getTables()) {
- newTableMap.put(entry, RSGroupInfo.getName());
- }
}
if (mutations.size() > 0) {
multiMutate(mutations);
}
- return newTableMap;
}
private synchronized void flushConfig() throws IOException {
@@ -500,8 +413,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
private synchronized void flushConfig(Map<String, RSGroupInfo> newGroupMap) throws IOException {
- Map<TableName, String> newTableMap;
-
// For offline mode persistence is still unavailable
// We're refreshing in-memory state but only for servers in default group
if (!isOnline()) {
@@ -516,7 +427,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
RSGroupInfo newDefaultGroup = newGroupMap.remove(RSGroupInfo.DEFAULT_GROUP);
if (!oldGroupMap.equals(newGroupMap) /* compare both tables and servers in other groups */ ||
!oldDefaultGroup.getTables().equals(newDefaultGroup.getTables())
- /* compare tables in default group */) {
+ /* compare tables in default group */) {
throw new IOException("Only servers in default group can be updated during offline mode");
}
@@ -533,11 +444,11 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
return;
}
- /* For online mode, persist to Zookeeper */
- newTableMap = flushConfigTable(newGroupMap);
+ /* For online mode, persist to hbase:rsgroup and Zookeeper */
+ flushConfigTable(newGroupMap);
// Make changes visible after having been persisted to the source of truth
- resetRSGroupAndTableMaps(newGroupMap, newTableMap);
+ resetRSGroupMap(newGroupMap);
try {
String groupBasePath =
@@ -575,11 +486,9 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
/**
* Make changes visible. Caller must be synchronized on 'this'.
*/
- private void resetRSGroupAndTableMaps(Map<String, RSGroupInfo> newRSGroupMap,
- Map<TableName, String> newTableMap) {
+ private void resetRSGroupMap(Map<String, RSGroupInfo> newRSGroupMap) {
// Make maps Immutable.
this.rsGroupMap = Collections.unmodifiableMap(newRSGroupMap);
- this.tableMap = Collections.unmodifiableMap(newTableMap);
}
/**
@@ -597,7 +506,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
return masterServices.getServerManager().getOnlineServersList();
}
LOG.debug("Reading online RS from zookeeper");
- List<ServerName> servers = new LinkedList<>();
+ List<ServerName> servers = new ArrayList<>();
try {
for (String el : ZKUtil.listChildrenNoWatch(watcher, watcher.getZNodePaths().rsZNode)) {
servers.add(ServerName.parseServerName(el));
@@ -633,7 +542,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// the rsGroupMap then writing it out.
private synchronized void updateDefaultServers(SortedSet<Address> servers) throws IOException {
RSGroupInfo info = rsGroupMap.get(RSGroupInfo.DEFAULT_GROUP);
- RSGroupInfo newInfo = new RSGroupInfo(info.getName(), servers, info.getTables());
+ RSGroupInfo newInfo = new RSGroupInfo(info.getName(), servers);
HashMap<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
newGroupMap.put(newInfo.getName(), newInfo);
flushConfig(newGroupMap);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
new file mode 100644
index 0000000..a08d236
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable
+ * law or agreed to in writing, software distributed under the License is distributed on an "AS IS"
+ * BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License
+ * for the specific language governing permissions and limitations under the License.
+ */
+package org.apache.hadoop.hbase.rsgroup;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Optional;
+import java.util.function.Predicate;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.ClusterSchema;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Helper class for RSGroup implementation
+ */
+@InterfaceAudience.Private
+final class RSGroupUtil {
+
+ private static final Logger LOG = LoggerFactory.getLogger(RSGroupUtil.class);
+
+ private RSGroupUtil() {
+ }
+
+ @FunctionalInterface
+ private interface GetRSGroup {
+ RSGroupInfo get(String groupName) throws IOException;
+ }
+
+ private static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, GetRSGroup getter,
+ TableName tableName) throws IOException {
+ TableDescriptor td = master.getTableDescriptors().get(tableName);
+ if (td == null) {
+ return Optional.empty();
+ }
+ Optional<String> optGroupNameOfTable = td.getRegionServerGroup();
+ if (optGroupNameOfTable.isPresent()) {
+ RSGroupInfo group = getter.get(optGroupNameOfTable.get());
+ if (group != null) {
+ return Optional.of(group);
+ }
+ }
+ ClusterSchema clusterSchema = master.getClusterSchema();
+ if (clusterSchema == null) {
+ if (TableName.isMetaTableName(tableName)) {
+ LOG.info("Can not get the namespace rs group config for meta table, since the" +
+ " meta table is not online yet, will use default group to assign meta first");
+ } else {
+ LOG.warn("ClusterSchema is null, can only use default rsgroup, should not happen?");
+ }
+ return Optional.empty();
+ }
+ NamespaceDescriptor nd = clusterSchema.getNamespace(tableName.getNamespaceAsString());
+ String groupNameOfNs = nd.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (groupNameOfNs == null) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(getter.get(groupNameOfNs));
+ }
+
+ /**
+ * Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
+ * from the {@link NamespaceDescriptor}. If still not present, return empty.
+ */
+ static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupInfoManager manager,
+ TableName tableName) throws IOException {
+ return getRSGroupInfo(master, manager::getRSGroup, tableName);
+ }
+
+ /**
+ * Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
+ * from the {@link NamespaceDescriptor}. If still not present, return empty.
+ */
+ static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupAdmin admin,
+ TableName tableName) throws IOException {
+ return getRSGroupInfo(master, admin::getRSGroupInfo, tableName);
+ }
+
+ /**
+ * Fill the tables field for {@link RSGroupInfo}, for backward compatibility.
+ */
+ @SuppressWarnings("deprecation")
+ static RSGroupInfo fillTables(RSGroupInfo rsGroupInfo, Collection<TableDescriptor> tds) {
+ RSGroupInfo newRsGroupInfo = new RSGroupInfo(rsGroupInfo);
+ Predicate<TableDescriptor> filter;
+ if (rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ filter = td -> {
+ Optional<String> optGroupName = td.getRegionServerGroup();
+ return !optGroupName.isPresent() || optGroupName.get().equals(RSGroupInfo.DEFAULT_GROUP);
+ };
+ } else {
+ filter = td -> {
+ Optional<String> optGroupName = td.getRegionServerGroup();
+ return optGroupName.isPresent() && optGroupName.get().equals(newRsGroupInfo.getName());
+ };
+ }
+ tds.stream().filter(filter).map(TableDescriptor::getTableName)
+ .forEach(newRsGroupInfo::addTable);
+ return newRsGroupInfo;
+ }
+}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement2.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement2.java
index 6dc3711..47337f9 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement2.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionPlacement2.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.master;
import static org.junit.Assert.assertTrue;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
@@ -26,7 +27,6 @@ import java.util.Map;
import java.util.Set;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseClassTestRule;
-import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.ServerName;
@@ -82,7 +82,7 @@ public class TestRegionPlacement2 {
}
@Test
- public void testFavoredNodesPresentForRoundRobinAssignment() throws HBaseIOException {
+ public void testFavoredNodesPresentForRoundRobinAssignment() throws IOException {
LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(TEST_UTIL.getConfiguration());
balancer.setMasterServices(TEST_UTIL.getMiniHBaseCluster().getMaster());
balancer.initialize();
@@ -143,7 +143,7 @@ public class TestRegionPlacement2 {
}
@Test
- public void testFavoredNodesPresentForRandomAssignment() throws HBaseIOException {
+ public void testFavoredNodesPresentForRandomAssignment() throws IOException {
LoadBalancer balancer = LoadBalancerFactory.getLoadBalancer(TEST_UTIL.getConfiguration());
balancer.setMasterServices(TEST_UTIL.getMiniHBaseCluster().getMaster());
balancer.initialize();
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
index 570bb3a..4c00bcf 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
@@ -28,6 +28,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
@@ -60,17 +61,13 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
public class RSGroupableBalancerTestBase {
static SecureRandom rand = new SecureRandom();
- static String[] groups = new String[] {RSGroupInfo.DEFAULT_GROUP, "dg2", "dg3", "dg4"};
+ static String[] groups = new String[] { RSGroupInfo.DEFAULT_GROUP, "dg2", "dg3", "dg4" };
static TableName table0 = TableName.valueOf("dt0");
- static TableName[] tables =
- new TableName[] { TableName.valueOf("dt1"),
- TableName.valueOf("dt2"),
- TableName.valueOf("dt3"),
- TableName.valueOf("dt4")};
+ static TableName[] tables = new TableName[] { TableName.valueOf("dt1"), TableName.valueOf("dt2"),
+ TableName.valueOf("dt3"), TableName.valueOf("dt4") };
static List<ServerName> servers;
static Map<String, RSGroupInfo> groupMap;
- static Map<TableName, String> tableMap = new HashMap<>();
- static List<TableDescriptor> tableDescs;
+ static Map<TableName, TableDescriptor> tableDescs;
int[] regionAssignment = new int[] { 2, 5, 7, 10, 4, 3, 1 };
static int regionId = 0;
@@ -113,20 +110,19 @@ public class RSGroupableBalancerTestBase {
/**
* All regions have an assignment.
*/
- protected void assertImmediateAssignment(List<RegionInfo> regions,
- List<ServerName> servers,
- Map<RegionInfo, ServerName> assignments)
- throws IOException {
+ protected void assertImmediateAssignment(List<RegionInfo> regions, List<ServerName> servers,
+ Map<RegionInfo, ServerName> assignments) throws IOException {
for (RegionInfo region : regions) {
assertTrue(assignments.containsKey(region));
ServerName server = assignments.get(region);
TableName tableName = region.getTable();
- String groupName = getMockedGroupInfoManager().getRSGroupOfTable(tableName);
+ String groupName =
+ tableDescs.get(tableName).getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
assertTrue(StringUtils.isNotEmpty(groupName));
RSGroupInfo gInfo = getMockedGroupInfoManager().getRSGroup(groupName);
assertTrue("Region is not correctly assigned to group servers.",
- gInfo.containsServer(server.getAddress()));
+ gInfo.containsServer(server.getAddress()));
}
}
@@ -169,16 +165,13 @@ public class RSGroupableBalancerTestBase {
ServerName oldAssignedServer = existing.get(r);
TableName tableName = r.getTable();
String groupName =
- getMockedGroupInfoManager().getRSGroupOfTable(tableName);
+ tableDescs.get(tableName).getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
assertTrue(StringUtils.isNotEmpty(groupName));
- RSGroupInfo gInfo = getMockedGroupInfoManager().getRSGroup(
- groupName);
- assertTrue(
- "Region is not correctly assigned to group servers.",
- gInfo.containsServer(currentServer.getAddress()));
- if (oldAssignedServer != null
- && onlineHostNames.contains(oldAssignedServer
- .getHostname())) {
+ RSGroupInfo gInfo = getMockedGroupInfoManager().getRSGroup(groupName);
+ assertTrue("Region is not correctly assigned to group servers.",
+ gInfo.containsServer(currentServer.getAddress()));
+ if (oldAssignedServer != null &&
+ onlineHostNames.contains(oldAssignedServer.getHostname())) {
// this region was previously assigned somewhere, and that
// host is still around, then the host must have been is a
// different group.
@@ -358,13 +351,12 @@ public class RSGroupableBalancerTestBase {
/**
* Construct group info, with each group having at least one server.
- *
* @param servers the servers
* @param groups the groups
* @return the map
*/
- protected static Map<String, RSGroupInfo> constructGroupInfo(
- List<ServerName> servers, String[] groups) {
+ protected static Map<String, RSGroupInfo> constructGroupInfo(List<ServerName> servers,
+ String[] groups) {
assertTrue(servers != null);
assertTrue(servers.size() >= groups.length);
int index = 0;
@@ -377,8 +369,7 @@ public class RSGroupableBalancerTestBase {
}
while (index < servers.size()) {
int grpIndex = rand.nextInt(groups.length);
- groupMap.get(groups[grpIndex]).addServer(
- servers.get(index).getAddress());
+ groupMap.get(groups[grpIndex]).addServer(servers.get(index).getAddress());
index++;
}
return groupMap;
@@ -389,29 +380,28 @@ public class RSGroupableBalancerTestBase {
* @param hasBogusTable there is a table that does not determine the group
* @return the list of table descriptors
*/
- protected static List<TableDescriptor> constructTableDesc(boolean hasBogusTable) {
- List<TableDescriptor> tds = Lists.newArrayList();
+ protected static Map<TableName, TableDescriptor> constructTableDesc(boolean hasBogusTable) {
+ Map<TableName, TableDescriptor> tds = new HashMap<>();
int index = rand.nextInt(groups.length);
for (int i = 0; i < tables.length; i++) {
- TableDescriptor htd = TableDescriptorBuilder.newBuilder(tables[i]).build();
int grpIndex = (i + index) % groups.length;
String groupName = groups[grpIndex];
- tableMap.put(tables[i], groupName);
- tds.add(htd);
+ TableDescriptor htd =
+ TableDescriptorBuilder.newBuilder(tables[i]).setRegionServerGroup(groupName).build();
+ tds.put(htd.getTableName(), htd);
}
if (hasBogusTable) {
- tableMap.put(table0, "");
- tds.add(TableDescriptorBuilder.newBuilder(table0).build());
+ tds.put(table0, TableDescriptorBuilder.newBuilder(table0).setRegionServerGroup("").build());
}
return tds;
}
protected static MasterServices getMockedMaster() throws IOException {
TableDescriptors tds = Mockito.mock(TableDescriptors.class);
- Mockito.when(tds.get(tables[0])).thenReturn(tableDescs.get(0));
- Mockito.when(tds.get(tables[1])).thenReturn(tableDescs.get(1));
- Mockito.when(tds.get(tables[2])).thenReturn(tableDescs.get(2));
- Mockito.when(tds.get(tables[3])).thenReturn(tableDescs.get(3));
+ Mockito.when(tds.get(tables[0])).thenReturn(tableDescs.get(tables[0]));
+ Mockito.when(tds.get(tables[1])).thenReturn(tableDescs.get(tables[1]));
+ Mockito.when(tds.get(tables[2])).thenReturn(tableDescs.get(tables[2]));
+ Mockito.when(tds.get(tables[3])).thenReturn(tableDescs.get(tables[3]));
MasterServices services = Mockito.mock(HMaster.class);
Mockito.when(services.getTableDescriptors()).thenReturn(tds);
AssignmentManager am = Mockito.mock(AssignmentManager.class);
@@ -430,13 +420,6 @@ public class RSGroupableBalancerTestBase {
Mockito.when(gm.listRSGroups()).thenReturn(
Lists.newLinkedList(groupMap.values()));
Mockito.when(gm.isOnline()).thenReturn(true);
- Mockito.when(gm.getRSGroupOfTable(Mockito.any()))
- .thenAnswer(new Answer<String>() {
- @Override
- public String answer(InvocationOnMock invocation) throws Throwable {
- return tableMap.get(invocation.getArgument(0));
- }
- });
return gm;
}
@@ -444,15 +427,16 @@ public class RSGroupableBalancerTestBase {
TableName tableName = null;
RSGroupInfoManager gm = getMockedGroupInfoManager();
RSGroupInfo groupOfServer = null;
- for(RSGroupInfo gInfo : gm.listRSGroups()){
- if(gInfo.containsServer(sn.getAddress())){
+ for (RSGroupInfo gInfo : gm.listRSGroups()) {
+ if (gInfo.containsServer(sn.getAddress())) {
groupOfServer = gInfo;
break;
}
}
- for(TableDescriptor desc : tableDescs){
- if(gm.getRSGroupOfTable(desc.getTableName()).endsWith(groupOfServer.getName())){
+ for (TableDescriptor desc : tableDescs.values()) {
+ Optional<String> optGroupName = desc.getRegionServerGroup();
+ if (optGroupName.isPresent() && optGroupName.get().endsWith(groupOfServer.getName())) {
tableName = desc.getTableName();
}
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
index 4b88201..2345a39 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
@@ -96,33 +96,30 @@ public class TestRSGroupBasedLoadBalancer extends RSGroupableBalancerTestBase {
/**
* Tests the bulk assignment used during cluster startup.
- *
- * Round-robin. Should yield a balanced cluster so same invariant as the
- * load balancer holds, all servers holding either floor(avg) or
- * ceiling(avg).
+ * <p/>
+ * Round-robin. Should yield a balanced cluster so same invariant as the load balancer holds, all
+ * servers holding either floor(avg) or ceiling(avg).
*/
@Test
public void testBulkAssignment() throws Exception {
List<RegionInfo> regions = randomRegions(25);
- Map<ServerName, List<RegionInfo>> assignments = loadBalancer
- .roundRobinAssignment(regions, servers);
- //test empty region/servers scenario
- //this should not throw an NPE
+ Map<ServerName, List<RegionInfo>> assignments =
+ loadBalancer.roundRobinAssignment(regions, servers);
+ // test empty region/servers scenario
+ // this should not throw an NPE
loadBalancer.roundRobinAssignment(regions, Collections.emptyList());
- //test regular scenario
+ // test regular scenario
assertTrue(assignments.keySet().size() == servers.size());
for (ServerName sn : assignments.keySet()) {
List<RegionInfo> regionAssigned = assignments.get(sn);
for (RegionInfo region : regionAssigned) {
TableName tableName = region.getTable();
String groupName =
- getMockedGroupInfoManager().getRSGroupOfTable(tableName);
+ tableDescs.get(tableName).getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
assertTrue(StringUtils.isNotEmpty(groupName));
- RSGroupInfo gInfo = getMockedGroupInfoManager().getRSGroup(
- groupName);
- assertTrue(
- "Region is not correctly assigned to group servers.",
- gInfo.containsServer(sn.getAddress()));
+ RSGroupInfo gInfo = getMockedGroupInfoManager().getRSGroup(groupName);
+ assertTrue("Region is not correctly assigned to group servers.",
+ gInfo.containsServer(sn.getAddress()));
}
}
ArrayListMultimap<String, ServerAndLoad> loadMap = convertToGroupBasedMap(assignments);
@@ -158,24 +155,25 @@ public class TestRSGroupBasedLoadBalancer extends RSGroupableBalancerTestBase {
onlineServers.addAll(servers);
List<RegionInfo> regions = randomRegions(25);
int bogusRegion = 0;
- for(RegionInfo region : regions){
- String group = tableMap.get(region.getTable());
- if("dg3".equals(group) || "dg4".equals(group)){
+ for (RegionInfo region : regions) {
+ String group = tableDescs.get(region.getTable()).getRegionServerGroup()
+ .orElse(RSGroupInfo.DEFAULT_GROUP);
+ if ("dg3".equals(group) || "dg4".equals(group)) {
bogusRegion++;
}
}
Set<Address> offlineServers = new HashSet<Address>();
offlineServers.addAll(groupMap.get("dg3").getServers());
offlineServers.addAll(groupMap.get("dg4").getServers());
- for(Iterator<ServerName> it = onlineServers.iterator(); it.hasNext();){
+ for (Iterator<ServerName> it = onlineServers.iterator(); it.hasNext();) {
ServerName server = it.next();
Address address = server.getAddress();
- if(offlineServers.contains(address)){
+ if (offlineServers.contains(address)) {
it.remove();
}
}
- Map<ServerName, List<RegionInfo>> assignments = loadBalancer
- .roundRobinAssignment(regions, onlineServers);
+ Map<ServerName, List<RegionInfo>> assignments =
+ loadBalancer.roundRobinAssignment(regions, onlineServers);
assertEquals(bogusRegion, assignments.get(LoadBalancer.BOGUS_SERVER_NAME).size());
}
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
index e588a7e..a4ae636 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
@@ -22,6 +22,7 @@ import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -32,7 +33,6 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.ClusterMetrics;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.RegionMetrics;
import org.apache.hadoop.hbase.ServerMetrics;
import org.apache.hadoop.hbase.ServerName;
@@ -98,7 +98,7 @@ public class TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal
* Test HBASE-20791
*/
@Test
- public void testBalanceCluster() throws HBaseIOException {
+ public void testBalanceCluster() throws IOException {
// mock cluster State
Map<ServerName, List<RegionInfo>> clusterState = new HashMap<ServerName, List<RegionInfo>>();
ServerName serverA = servers.get(0);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
index 27511e3..7471458 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
@@ -29,7 +29,6 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.SortedSet;
-
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.MiniHBaseCluster;
import org.apache.hadoop.hbase.NamespaceDescriptor;
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
index 4b4097f..076362e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
@@ -21,7 +21,6 @@ import static org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.DEFAULT_MAX_RET
import static org.apache.hadoop.hbase.util.Threads.sleep;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -408,7 +407,9 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
assertTrue(newGroupTables.contains(tableName));
// verify that all region still assgin on targetServer
- Assert.assertEquals(5, getTableServerRegionMap().get(tableName).get(targetServer).size());
+ // TODO: uncomment after we reimplement moveServersAndTables, now the implementation is
+ // moveServers first and then moveTables, so the region will be moved to other region servers.
+ // Assert.assertEquals(5, getTableServerRegionMap().get(tableName).get(targetServer).size());
assertTrue(observer.preMoveServersAndTables);
assertTrue(observer.postMoveServersAndTables);
@@ -503,61 +504,6 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
});
}
- @Test
- public void testFailedMoveBeforeRetryExhaustedWhenMoveTable() throws Exception {
- final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()), 1);
- Pair<ServerName, RegionStateNode> gotPair = createTableWithRegionSplitting(newGroup,
- 5);
-
- // move table to group
- Thread t2 = new Thread(() -> {
- LOG.info("thread2 start running, to move regions");
- try {
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
- } catch (IOException e) {
- LOG.error("move server error", e);
- }
- });
- t2.start();
-
- // start thread to recover region state
- final ServerName ss = gotPair.getFirst();
- final RegionStateNode rsn = gotPair.getSecond();
- AtomicBoolean changed = new AtomicBoolean(false);
-
- Thread t1 = recoverRegionStateThread(ss, server -> {
- List<RegionInfo> regions = master.getAssignmentManager().getRegionsOnServer(ss);
- List<RegionInfo> tableRegions = new ArrayList<>();
- for (RegionInfo regionInfo : regions) {
- if (regionInfo.getTable().equals(tableName)) {
- tableRegions.add(regionInfo);
- }
- }
- return tableRegions;
- }, rsn, changed);
- t1.start();
-
- t1.join();
- t2.join();
-
- TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
- @Override
- public boolean evaluate() {
- if (changed.get()) {
- boolean serverHasTableRegions = false;
- for (RegionInfo regionInfo : master.getAssignmentManager().getRegionsOnServer(ss)) {
- if (regionInfo.getTable().equals(tableName)) {
- serverHasTableRegions = true;
- break;
- }
- }
- return !serverHasTableRegions && !rsn.getRegionLocation().equals(ss);
- }
- return false;
- }
- });
- }
-
private <T> Thread recoverRegionStateThread(T owner, Function<T, List<RegionInfo>> getRegions,
RegionStateNode rsn, AtomicBoolean changed){
return new Thread(() -> {
@@ -653,50 +599,6 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
}
@Test
- public void testFailedMoveTablesAndRepair() throws Exception{
- // This UT calls moveTables() twice to test the idempotency of it.
- // The first time, movement fails because a region is made in SPLITTING state
- // which will not be moved.
- // The second time, the region state is OPEN and check if all
- // regions on target group servers after the call.
- final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()), 1);
- Iterator iterator = newGroup.getServers().iterator();
- Address newGroupServer1 = (Address) iterator.next();
-
- // create table
- // randomly set a region state to SPLITTING to make move abort
- Pair<ServerName, RegionStateNode> gotPair = createTableWithRegionSplitting(newGroup,
- new Random().nextInt(8) + 4);
- RegionStateNode rsn = gotPair.getSecond();
-
- // move table to newGroup and check regions
- try {
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
- fail("should get IOException when retry exhausted but there still exists failed moved "
- + "regions");
- }catch (Exception e){
- assertTrue(e.getMessage().contains(
- gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
- }
- for(RegionInfo regionInfo : master.getAssignmentManager().getAssignedRegions()){
- if (regionInfo.getTable().equals(tableName) && regionInfo.equals(rsn.getRegionInfo())) {
- assertNotEquals(master.getAssignmentManager().getRegionStates()
- .getRegionServerOfRegion(regionInfo).getAddress(), newGroupServer1);
- }
- }
-
- // retry move table to newGroup and check if all regions are corrected
- rsn.setState(RegionState.State.OPEN);
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
- for(RegionInfo regionInfo : master.getAssignmentManager().getAssignedRegions()){
- if (regionInfo.getTable().equals(tableName)) {
- assertEquals(master.getAssignmentManager().getRegionStates()
- .getRegionServerOfRegion(regionInfo).getAddress(), newGroupServer1);
- }
- }
- }
-
- @Test
public void testFailedMoveServersAndRepair() throws Exception{
// This UT calls moveServers() twice to test the idempotency of it.
// The first time, movement fails because a region is made in SPLITTING state
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
index 67f5c7e..8d10850 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
@@ -45,8 +45,6 @@ import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
-
@Category({ MediumTests.class })
public class TestRSGroupsBalance extends TestRSGroupsBase {
@@ -153,19 +151,21 @@ public class TestRSGroupsBalance extends TestRSGroupsBase {
@Test
public void testMisplacedRegions() throws Exception {
- final TableName tableName = TableName.valueOf(tablePrefix + "_testMisplacedRegions");
- LOG.info("testMisplacedRegions");
+ String namespace = tablePrefix + "_" + name.getMethodName();
+ TEST_UTIL.getAdmin().createNamespace(NamespaceDescriptor.create(namespace).build());
+ final TableName tableName =
+ TableName.valueOf(namespace, tablePrefix + "_" + name.getMethodName());
+ LOG.info(name.getMethodName());
- final RSGroupInfo RSGroupInfo = addGroup("testMisplacedRegions", 1);
+ final RSGroupInfo rsGroupInfo = addGroup(name.getMethodName(), 1);
TEST_UTIL.createMultiRegionTable(tableName, new byte[] { 'f' }, 15);
TEST_UTIL.waitUntilAllRegionsAssigned(tableName);
-
- rsGroupAdminEndpoint.getGroupInfoManager().moveTables(Sets.newHashSet(tableName),
- RSGroupInfo.getName());
+ TEST_UTIL.getAdmin().modifyNamespace(NamespaceDescriptor.create(namespace)
+ .addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, rsGroupInfo.getName()).build());
admin.balancerSwitch(true, true);
- assertTrue(rsGroupAdmin.balanceRSGroup(RSGroupInfo.getName()));
+ assertTrue(rsGroupAdmin.balanceRSGroup(rsGroupInfo.getName()));
admin.balancerSwitch(false, true);
assertTrue(observer.preBalanceRSGroupCalled);
assertTrue(observer.postBalanceRSGroupCalled);
@@ -174,7 +174,7 @@ public class TestRSGroupsBalance extends TestRSGroupsBase {
@Override
public boolean evaluate() throws Exception {
ServerName serverName =
- ServerName.valueOf(RSGroupInfo.getServers().iterator().next().toString(), 1);
+ ServerName.valueOf(rsGroupInfo.getServers().iterator().next().toString(), 1);
return admin.getConnection().getAdmin().getRegions(serverName).size() == 15;
}
});
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
index 0e4fb34..b91cd5e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
@@ -74,7 +74,7 @@ public abstract class TestRSGroupsBase {
protected static HBaseTestingUtility TEST_UTIL;
protected static Admin admin;
protected static HBaseCluster cluster;
- protected static RSGroupAdmin rsGroupAdmin;
+ protected static RSGroupAdminClient rsGroupAdmin;
protected static HMaster master;
protected boolean INIT = false;
protected static RSGroupAdminEndpoint rsGroupAdminEndpoint;
@@ -188,8 +188,8 @@ public abstract class TestRSGroupsBase {
RSGroupInfo defaultInfo = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.addRSGroup(groupName);
Set<Address> set = new HashSet<>();
- for(Address server: defaultInfo.getServers()) {
- if(set.size() == serverCount) {
+ for (Address server : defaultInfo.getServers()) {
+ if (set.size() == serverCount) {
break;
}
set.add(server);
@@ -222,7 +222,7 @@ public abstract class TestRSGroupsBase {
}
protected void deleteGroups() throws IOException {
- RSGroupAdmin groupAdmin = new RSGroupAdminClient(TEST_UTIL.getConnection());
+ RSGroupAdminClient groupAdmin = new RSGroupAdminClient(TEST_UTIL.getConnection());
for(RSGroupInfo group: groupAdmin.listRSGroups()) {
if(!group.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
groupAdmin.moveTables(group.getTables(), RSGroupInfo.DEFAULT_GROUP);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
index 60887e4..d3577f2 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hbase.rsgroup;
-import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import org.apache.hadoop.hbase.HBaseClassTestRule;
@@ -113,7 +112,7 @@ public class TestRSGroupsOfflineMode {
final HRegionServer groupRS = ((MiniHBaseCluster) cluster).getRegionServer(1);
final HRegionServer failoverRS = ((MiniHBaseCluster) cluster).getRegionServer(2);
String newGroup = "my_group";
- RSGroupAdmin groupAdmin = new RSGroupAdminClient(TEST_UTIL.getConnection());
+ RSGroupAdminClient groupAdmin = new RSGroupAdminClient(TEST_UTIL.getConnection());
groupAdmin.addRSGroup(newGroup);
if (master.getAssignmentManager().getRegionStates().getRegionAssignments()
.containsValue(failoverRS.getServerName())) {
@@ -168,9 +167,6 @@ public class TestRSGroupsOfflineMode {
.getMasterCoprocessorHost().findCoprocessor(RSGroupAdminEndpoint.class).getGroupInfoManager();
// Make sure balancer is in offline mode, since this is what we're testing.
assertFalse(groupMgr.isOnline());
- // Verify the group affiliation that's loaded from ZK instead of tables.
- assertEquals(newGroup, groupMgr.getRSGroupOfTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME));
- assertEquals(RSGroupInfo.DEFAULT_GROUP, groupMgr.getRSGroupOfTable(failoverTable));
// Kill final regionserver to see the failover happens for all tables except GROUP table since
// it's group does not have any online RS.
killRS.stop("die");
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
index fcaf1a7..a8cd277 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
@@ -17,17 +17,26 @@
*/
package org.apache.hadoop.hbase.rsgroup;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
import java.io.ByteArrayInputStream;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
@@ -37,22 +46,20 @@ import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
import org.apache.hadoop.hbase.zookeeper.ZNodePaths;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
-import org.junit.Assert;
import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
@InterfaceAudience.Private
-public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
- private Table table;
+public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
+ private Connection conn;
private ZKWatcher zkw;
- private RSGroupAdmin wrapped;
+ private RSGroupAdminClient wrapped;
- public VerifyingRSGroupAdminClient(RSGroupAdmin RSGroupAdmin, Configuration conf)
+ public VerifyingRSGroupAdminClient(RSGroupAdminClient RSGroupAdmin, Configuration conf)
throws IOException {
wrapped = RSGroupAdmin;
- table = ConnectionFactory.createConnection(conf)
- .getTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME);
+ conn = ConnectionFactory.createConnection(conf);
zkw = new ZKWatcher(conf, this.getClass().getSimpleName(), null);
}
@@ -121,31 +128,41 @@ public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
public void verify() throws IOException {
Map<String, RSGroupInfo> groupMap = Maps.newHashMap();
Set<RSGroupInfo> zList = Sets.newHashSet();
-
- for (Result result : table.getScanner(new Scan())) {
- RSGroupProtos.RSGroupInfo proto =
- RSGroupProtos.RSGroupInfo.parseFrom(
- result.getValue(
- RSGroupInfoManagerImpl.META_FAMILY_BYTES,
- RSGroupInfoManagerImpl.META_QUALIFIER_BYTES));
- groupMap.put(proto.getName(), ProtobufUtil.toGroupInfo(proto));
+ List<TableDescriptor> tds = new ArrayList<>();
+ try (Admin admin = conn.getAdmin()) {
+ tds.addAll(admin.listTableDescriptors());
+ tds.addAll(admin.listTableDescriptorsByNamespace(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME));
+ }
+ try (Table table = conn.getTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME);
+ ResultScanner scanner = table.getScanner(new Scan())) {
+ for (;;) {
+ Result result = scanner.next();
+ if (result == null) {
+ break;
+ }
+ RSGroupProtos.RSGroupInfo proto = RSGroupProtos.RSGroupInfo.parseFrom(result.getValue(
+ RSGroupInfoManagerImpl.META_FAMILY_BYTES, RSGroupInfoManagerImpl.META_QUALIFIER_BYTES));
+ RSGroupInfo rsGroupInfo = ProtobufUtil.toGroupInfo(proto);
+ groupMap.put(proto.getName(), RSGroupUtil.fillTables(rsGroupInfo, tds));
+ }
}
- Assert.assertEquals(Sets.newHashSet(groupMap.values()),
- Sets.newHashSet(wrapped.listRSGroups()));
+ assertEquals(Sets.newHashSet(groupMap.values()), Sets.newHashSet(wrapped.listRSGroups()));
try {
String groupBasePath = ZNodePaths.joinZNode(zkw.getZNodePaths().baseZNode, "rsgroup");
- for(String znode: ZKUtil.listChildrenNoWatch(zkw, groupBasePath)) {
+ for (String znode : ZKUtil.listChildrenNoWatch(zkw, groupBasePath)) {
byte[] data = ZKUtil.getData(zkw, ZNodePaths.joinZNode(groupBasePath, znode));
- if(data.length > 0) {
+ if (data.length > 0) {
ProtobufUtil.expectPBMagicPrefix(data);
- ByteArrayInputStream bis = new ByteArrayInputStream(
- data, ProtobufUtil.lengthOfPBMagic(), data.length);
- zList.add(ProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis)));
+ ByteArrayInputStream bis =
+ new ByteArrayInputStream(data, ProtobufUtil.lengthOfPBMagic(), data.length);
+ RSGroupInfo rsGroupInfo =
+ ProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis));
+ zList.add(RSGroupUtil.fillTables(rsGroupInfo, tds));
}
}
- Assert.assertEquals(zList.size(), groupMap.size());
- for(RSGroupInfo RSGroupInfo : zList) {
- Assert.assertTrue(groupMap.get(RSGroupInfo.getName()).equals(RSGroupInfo));
+ assertEquals(zList.size(), groupMap.size());
+ for (RSGroupInfo rsGroupInfo : zList) {
+ assertTrue(groupMap.get(rsGroupInfo.getName()).equals(rsGroupInfo));
}
} catch (KeeperException e) {
throw new IOException("ZK verification failed", e);
[hbase] 14/14: HBASE-22932 Add rs group management methods in Admin
and AsyncAdmin (#657)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit eb83d580aec5660d26aeec91d7c6fcb100028c0f
Author: linkaline <li...@gmail.com>
AuthorDate: Tue Nov 5 14:15:51 2019 +0800
HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
Signed-off-by: Duo Zhang <zh...@apache.org>
---
.../java/org/apache/hadoop/hbase/client/Admin.java | 78 ++
.../hadoop/hbase/client/AdminOverAsyncAdmin.java | 52 +
.../org/apache/hadoop/hbase/client/AsyncAdmin.java | 80 +-
.../hadoop/hbase/client/AsyncHBaseAdmin.java | 51 +
.../hadoop/hbase/client/RawAsyncHBaseAdmin.java | 209 +++-
.../hbase/shaded/protobuf/RequestConverter.java | 37 +-
.../hbase/rsgroup/IntegrationTestRSGroup.java | 4 +-
.../src/main/protobuf/Master.proto | 19 +
.../org/apache/hadoop/hbase/master/HMaster.java | 2 +-
.../hadoop/hbase/master/MasterCoprocessorHost.java | 6 +-
.../hadoop/hbase/master/MasterRpcServices.java | 171 +++-
.../apache/hadoop/hbase/master/MasterServices.java | 9 +
.../hbase/rsgroup/DisabledRSGroupInfoManager.java | 13 +-
.../apache/hadoop/hbase/rsgroup/RSGroupAdmin.java | 4 +
.../hadoop/hbase/rsgroup/RSGroupAdminClient.java | 1041 ++++++++++++++++++--
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 25 +-
.../hadoop/hbase/rsgroup/RSGroupAdminServer.java | 495 ----------
.../hbase/rsgroup/RSGroupAdminServiceImpl.java | 83 +-
.../hadoop/hbase/rsgroup/RSGroupInfoManager.java | 20 +-
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 444 ++++++++-
.../hbase/rsgroup/RSGroupMajorCompactionTTL.java | 7 +-
.../apache/hadoop/hbase/rsgroup/RSGroupUtil.java | 8 +-
.../hbase/security/access/AccessController.java | 81 ++
.../hbase/master/MockNoopMasterServices.java | 5 +
.../hbase/rsgroup/TestMigrateRSGroupInfo.java | 28 +-
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java | 131 ++-
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java | 130 +--
.../hadoop/hbase/rsgroup/TestRSGroupsBalance.java | 15 +-
.../hadoop/hbase/rsgroup/TestRSGroupsBase.java | 97 +-
.../hadoop/hbase/rsgroup/TestRSGroupsBasics.java | 43 +-
.../hadoop/hbase/rsgroup/TestRSGroupsKillRS.java | 26 +-
.../hbase/rsgroup/TestRSGroupsOfflineMode.java | 6 +-
.../hadoop/hbase/rsgroup/TestRSGroupsWithACL.java | 47 +-
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 29 +-
.../hadoop/hbase/thrift2/client/ThriftAdmin.java | 52 +
35 files changed, 2577 insertions(+), 971 deletions(-)
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
index 5419c16..2b4eaf8 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
@@ -45,6 +45,7 @@ import org.apache.hadoop.hbase.client.replication.ReplicationPeerConfigUtil;
import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshotView;
@@ -53,6 +54,7 @@ import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.UserPermission;
@@ -2262,4 +2264,80 @@ public interface Admin extends Abortable, Closeable {
*/
boolean isSnapshotCleanupEnabled() throws IOException;
+ /**
+ * Creates a new RegionServer group with the given name
+ * @param groupName the name of the group
+ * @throws IOException if a remote or network exception occurs
+ */
+ void addRSGroup(String groupName) throws IOException;
+
+ /**
+ * Get group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ * @throws IOException if a remote or network exception occurs
+ */
+ RSGroupInfo getRSGroup(String groupName) throws IOException;
+
+ /**
+ * Get group info for the given hostPort
+ * @param hostPort HostPort to get RSGroupInfo for
+ * @throws IOException if a remote or network exception occurs
+ */
+ RSGroupInfo getRSGroup(Address hostPort) throws IOException;
+
+ /**
+ * Get group info for the given table
+ * @param tableName table name to get RSGroupInfo for
+ * @throws IOException if a remote or network exception occurs
+ */
+ RSGroupInfo getRSGroup(TableName tableName) throws IOException;
+
+ /**
+ * Lists current set of RegionServer groups
+ * @throws IOException if a remote or network exception occurs
+ */
+ List<RSGroupInfo> listRSGroups() throws IOException;
+
+ /**
+ * Remove RegionServer group associated with the given name
+ * @param groupName the group name
+ * @throws IOException if a remote or network exception occurs
+ */
+ void removeRSGroup(String groupName) throws IOException;
+
+ /**
+ * Remove decommissioned servers from group
+ * 1. Sometimes we may find the server aborted due to some hardware failure and we must offline
+ * the server for repairing. Or we need to move some servers to join other clusters.
+ * So we need to remove these servers from the group.
+ * 2. Dead/recovering/live servers will be disallowed.
+ * @param servers set of servers to remove
+ * @throws IOException if a remote or network exception occurs
+ */
+ void removeRSGroup(Set<Address> servers) throws IOException;
+
+ /**
+ * Move given set of servers to the specified target RegionServer group
+ * @param servers set of servers to move
+ * @param targetGroup the group to move servers to
+ * @throws IOException if a remote or network exception occurs
+ */
+ void moveToRSGroup(Set<Address> servers, String targetGroup) throws IOException;
+
+ /**
+ * Set the RegionServer group for tables
+ * @param tables tables to set group for
+ * @param groupName group name for tables
+ * @throws IOException if a remote or network exception occurs
+ */
+ void setRSGroup(Set<TableName> tables, String groupName) throws IOException;
+
+ /**
+ * Balance regions in the given RegionServer group
+ * @param groupName the group name
+ * @return boolean Whether balance ran or not
+ * @throws IOException if a remote or network exception occurs
+ */
+ boolean balanceRSGroup(String groupName) throws IOException;
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
index f1f5b2a..060170c 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
@@ -49,6 +49,7 @@ import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshotView;
@@ -56,6 +57,7 @@ import org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.UserPermission;
@@ -960,4 +962,54 @@ class AdminOverAsyncAdmin implements Admin {
return get(admin.isSnapshotCleanupEnabled());
}
+ @Override
+ public RSGroupInfo getRSGroup(String groupName) throws IOException {
+ return get(admin.getRSGroup(groupName));
+ }
+
+ @Override
+ public void moveToRSGroup(Set<Address> servers, String groupName) throws IOException {
+ get(admin.moveToRSGroup(servers, groupName));
+ }
+
+ @Override
+ public void addRSGroup(String groupName) throws IOException {
+ get(admin.addRSGroup(groupName));
+ }
+
+ @Override
+ public void removeRSGroup(String groupName) throws IOException {
+ get(admin.removeRSGroup(groupName));
+ }
+
+ @Override
+ public boolean balanceRSGroup(String groupName) throws IOException {
+ return get(admin.balanceRSGroup(groupName));
+ }
+
+ @Override
+ public List<RSGroupInfo> listRSGroups() throws IOException {
+ return get(admin.listRSGroups());
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(Address hostPort) throws IOException {
+ return get(admin.getRSGroup(hostPort));
+ }
+
+ @Override
+ public void removeRSGroup(Set<Address> servers) throws IOException {
+ get(admin.removeRSGroup(servers));
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(TableName tableName) throws IOException {
+ return get(admin.getRSGroup(tableName));
+ }
+
+ @Override
+ public void setRSGroup(Set<TableName> tables, String groupName) throws IOException {
+ get(admin.setRSGroup(tables, groupName));
+ }
+
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
index a53646d..12926cc 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
@@ -40,12 +40,14 @@ import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshotView;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.UserPermission;
@@ -1381,7 +1383,7 @@ public interface AsyncAdmin {
* @param newTableName name of the new table where the table will be created
* @param preserveSplits True if the splits should be preserved
*/
- CompletableFuture<Void> cloneTableSchema(final TableName tableName,
+ CompletableFuture<Void> cloneTableSchema(final TableName tableName,
final TableName newTableName, final boolean preserveSplits);
/**
@@ -1507,4 +1509,80 @@ public interface AsyncAdmin {
*/
CompletableFuture<Boolean> isSnapshotCleanupEnabled();
+ /**
+ * Creates a new RegionServer group with the given name
+ * @param groupName the name of the group
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Void> addRSGroup(String groupName);
+
+ /**
+ * Get group info for the given group name
+ * @param groupName the group name
+ * @return group info
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<RSGroupInfo> getRSGroup(String groupName);
+
+ /**
+ * Get group info for the given hostPort
+ * @param hostPort HostPort to get RSGroupInfo for
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<RSGroupInfo> getRSGroup(Address hostPort);
+
+ /**
+ * Get group info for the given table
+ * @param tableName table name to get RSGroupInfo for
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<RSGroupInfo> getRSGroup(TableName tableName);
+
+ /**
+ * Lists current set of RegionServer groups
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<List<RSGroupInfo>> listRSGroups();
+
+ /**
+ * Remove RegionServer group associated with the given name
+ * @param groupName the group name
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Void> removeRSGroup(String groupName);
+
+ /**
+ * Remove decommissioned servers from group
+ * 1. Sometimes we may find the server aborted due to some hardware failure and we must offline
+ * the server for repairing. Or we need to move some servers to join other clusters.
+ * So we need to remove these servers from the group.
+ * 2. Dead/recovering/live servers will be disallowed.
+ * @param servers set of servers to remove
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Void> removeRSGroup(Set<Address> servers);
+
+ /**
+ * Move given set of servers to the specified target RegionServer group
+ * @param servers set of servers to move
+ * @param groupName the group to move servers to
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Void> moveToRSGroup(Set<Address> servers, String groupName);
+
+ /**
+ * Set the RegionServer group for tables
+ * @param tables tables to set group for
+ * @param groupName group name for tables
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Void> setRSGroup(Set<TableName> tables, String groupName);
+
+ /**
+ * Balance regions in the given RegionServer group
+ * @param groupName the group name
+ * @return boolean Whether balance ran or not
+ * @throws IOException if a remote or network exception occurs
+ */
+ CompletableFuture<Boolean> balanceRSGroup(String groupName);
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java
index ffb484c..459c225 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java
@@ -36,12 +36,14 @@ import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshot;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.UserPermission;
@@ -838,4 +840,53 @@ class AsyncHBaseAdmin implements AsyncAdmin {
return wrap(rawAdmin.isSnapshotCleanupEnabled());
}
+ public CompletableFuture<RSGroupInfo> getRSGroup(String groupName) {
+ return wrap(rawAdmin.getRSGroup(groupName));
+ }
+
+ @Override
+ public CompletableFuture<Void> moveToRSGroup(Set<Address> servers, String groupName) {
+ return wrap(rawAdmin.moveToRSGroup(servers, groupName));
+ }
+
+ @Override
+ public CompletableFuture<Void> addRSGroup(String groupName) {
+ return wrap(rawAdmin.addRSGroup(groupName));
+ }
+
+ @Override
+ public CompletableFuture<Void> removeRSGroup(String groupName) {
+ return wrap(rawAdmin.removeRSGroup(groupName));
+ }
+
+ @Override
+ public CompletableFuture<Boolean> balanceRSGroup(String groupName) {
+ return wrap(rawAdmin.balanceRSGroup(groupName));
+ }
+
+ @Override
+ public CompletableFuture<List<RSGroupInfo>> listRSGroups() {
+ return wrap(rawAdmin.listRSGroups());
+ }
+
+ @Override
+ public CompletableFuture<RSGroupInfo> getRSGroup(Address hostPort) {
+ return wrap(rawAdmin.getRSGroup(hostPort));
+ }
+
+ @Override
+ public CompletableFuture<Void> removeRSGroup(Set<Address> servers) {
+ return wrap(rawAdmin.removeRSGroup(servers));
+ }
+
+ @Override
+ public CompletableFuture<RSGroupInfo> getRSGroup(TableName tableName) {
+ return wrap(rawAdmin.getRSGroup(tableName));
+ }
+
+ @Override
+ public CompletableFuture<Void> setRSGroup(Set<TableName> tables, String groupName) {
+ return wrap(rawAdmin.setRSGroup(tables, groupName));
+ }
+
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
index 8c6ec0d..399ae28 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
@@ -78,6 +78,7 @@ import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.ipc.HBaseRpcController;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.QuotaTableUtil;
@@ -86,6 +87,7 @@ import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.ShadedAccessControlUtil;
@@ -206,8 +208,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsProcedur
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsProcedureDoneResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsRpcThrottleEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsRpcThrottleEnabledResponse;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .IsSnapshotCleanupEnabledResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotCleanupEnabledResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotDoneRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotDoneResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSplitOrMergeEnabledRequest;
@@ -258,8 +259,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetNormali
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetNormalizerRunningResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetQuotaRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetQuotaResponse;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .SetSnapshotCleanupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSnapshotCleanupResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSplitOrMergeEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSplitOrMergeEnabledResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ShutdownRequest;
@@ -285,6 +285,18 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuo
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaRegionSizesResponse.RegionSizes;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaSnapshotsRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaSnapshotsResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.AddRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.AddRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.MoveServersResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveServersResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationProtos.AddReplicationPeerRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationProtos.AddReplicationPeerResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationProtos.DisableReplicationPeerRequest;
@@ -3867,24 +3879,189 @@ class RawAsyncHBaseAdmin implements AsyncAdmin {
@Override
public CompletableFuture<Boolean> snapshotCleanupSwitch(final boolean on,
final boolean sync) {
- return this.<Boolean>newMasterCaller()
- .action((controller, stub) -> this
- .call(controller, stub,
- RequestConverter.buildSetSnapshotCleanupRequest(on, sync),
- MasterService.Interface::switchSnapshotCleanup,
- SetSnapshotCleanupResponse::getPrevSnapshotCleanup))
- .call();
+ return this.<Boolean>newMasterCaller().action((controller, stub) -> this
+ .call(controller, stub, RequestConverter.buildSetSnapshotCleanupRequest(on, sync),
+ MasterService.Interface::switchSnapshotCleanup,
+ SetSnapshotCleanupResponse::getPrevSnapshotCleanup)).call();
}
@Override
public CompletableFuture<Boolean> isSnapshotCleanupEnabled() {
- return this.<Boolean>newMasterCaller()
+ return this.<Boolean>newMasterCaller().action((controller, stub) -> this
+ .call(controller, stub, RequestConverter.buildIsSnapshotCleanupEnabledRequest(),
+ MasterService.Interface::isSnapshotCleanupEnabled,
+ IsSnapshotCleanupEnabledResponse::getEnabled)).call();
+ }
+
+ @Override
+ public CompletableFuture<Void> moveToRSGroup(Set<Address> servers, String groupName) {
+ return this.<Void> newMasterCaller()
+ .action((controller, stub) -> this.
+ <MoveServersRequest, MoveServersResponse, Void> call(controller, stub,
+ RequestConverter.buildMoveServersRequest(servers, groupName),
+ (s, c, req, done) -> s.moveServers(c, req, done), resp -> null))
+ .call();
+ }
+
+ @Override
+ public CompletableFuture<Void> addRSGroup(String groupName) {
+ return this.<Void> newMasterCaller()
+ .action(((controller, stub) -> this.
+ <AddRSGroupRequest, AddRSGroupResponse, Void> call(controller, stub,
+ AddRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.addRSGroup(c, req, done), resp -> null)))
+ .call();
+ }
+
+ @Override
+ public CompletableFuture<Void> removeRSGroup(String groupName) {
+ return this.<Void> newMasterCaller()
+ .action((controller, stub) -> this.
+ <RemoveRSGroupRequest, RemoveRSGroupResponse, Void> call(controller, stub,
+ RemoveRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.removeRSGroup(c, req, done), resp -> null))
+ .call();
+ }
+
+ @Override
+ public CompletableFuture<Boolean> balanceRSGroup(String groupName) {
+ return this.<Boolean> newMasterCaller()
+ .action((controller, stub) -> this.
+ <BalanceRSGroupRequest, BalanceRSGroupResponse, Boolean> call(controller, stub,
+ BalanceRSGroupRequest.newBuilder().setRSGroupName(groupName).build(),
+ (s, c, req, done) -> s.balanceRSGroup(c, req, done), resp -> resp.getBalanceRan()))
+ .call();
+ }
+
+ @Override
+ public CompletableFuture<List<RSGroupInfo>> listRSGroups() {
+ return this.<List<RSGroupInfo>> newMasterCaller()
.action((controller, stub) -> this
- .call(controller, stub,
- RequestConverter.buildIsSnapshotCleanupEnabledRequest(),
- MasterService.Interface::isSnapshotCleanupEnabled,
- IsSnapshotCleanupEnabledResponse::getEnabled))
+ .<ListRSGroupInfosRequest, ListRSGroupInfosResponse, List<RSGroupInfo>> call(
+ controller, stub, ListRSGroupInfosRequest.getDefaultInstance(),
+ (s, c, req, done) -> s.listRSGroupInfos(c, req, done),
+ resp -> resp.getRSGroupInfoList().stream()
+ .map(r -> ProtobufUtil.toGroupInfo(r))
+ .collect(Collectors.toList())))
.call();
}
+ @Override
+ public CompletableFuture<RSGroupInfo> getRSGroup(Address hostPort) {
+ CompletableFuture<RSGroupInfo> future = new CompletableFuture<>();
+ addListener(listRSGroups(), (groups, err) -> {
+ if (err != null) {
+ future.completeExceptionally(err);
+ return;
+ }
+ for (RSGroupInfo rsGroupInfo : groups) {
+ if (rsGroupInfo.getServers().contains(hostPort)){
+ future.complete(rsGroupInfo);
+ return;
+ }
+ }
+ future.complete(null);
+ });
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<Void> removeRSGroup(Set<Address> servers) {
+ return this.<Void> newMasterCaller()
+ .action((controller, stub) -> this.
+ <RemoveServersRequest, RemoveServersResponse, Void> call(controller, stub,
+ RequestConverter.buildRemoveServersRequest(servers),
+ (s, c, req, done) -> s.removeServers(c, req, done), resp -> null))
+ .call();
+ }
+
+ @Override
+ public CompletableFuture<Void> setRSGroup(Set<TableName> tables, String groupName) {
+ CompletableFuture<Void> future = new CompletableFuture<>();
+ for (TableName tableName : tables) {
+ addListener(tableExists(tableName), (exist, err) -> {
+ if (err != null) {
+ future.completeExceptionally(err);
+ return;
+ }
+ if (!exist) {
+ future.completeExceptionally(new TableNotFoundException(tableName));
+ return;
+ }
+ });
+ }
+ addListener(listTableDescriptors(new ArrayList<>(tables)), ((tableDescriptions, err) -> {
+ if (err != null) {
+ future.completeExceptionally(err);
+ return;
+ }
+ if (tableDescriptions == null || tableDescriptions.isEmpty()) {
+ future.complete(null);
+ return;
+ }
+ List<TableDescriptor> newTableDescriptors = new ArrayList<>();
+ for (TableDescriptor td : tableDescriptions) {
+ newTableDescriptors
+ .add(TableDescriptorBuilder.newBuilder(td).setRegionServerGroup(groupName).build());
+ }
+ addListener(CompletableFuture.allOf(
+ newTableDescriptors.stream().map(this::modifyTable).toArray(CompletableFuture[]::new)),
+ (v, e) -> {
+ if (e != null) {
+ future.completeExceptionally(e);
+ } else {
+ future.complete(v);
+ }
+ });
+ }));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<RSGroupInfo> getRSGroup(TableName table) {
+ CompletableFuture<RSGroupInfo> future = new CompletableFuture<>();
+ addListener(getDescriptor(table), (td, err) -> {
+ if (err != null) {
+ // return null instead of err to keep compatible with old semantics
+ // todo: need to change both this and UTs
+ future.complete(null);
+ return;
+ }
+ addListener(listRSGroups(), (groups, err2) -> {
+ if (err2 != null) {
+ future.completeExceptionally(err2);
+ return;
+ }
+ for (RSGroupInfo rsGroupInfo : groups) {
+ if (rsGroupInfo.getTables().contains(table)) {
+ future.complete(rsGroupInfo);
+ return;
+ }
+ }
+ future.complete(null);
+ });
+ });
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<RSGroupInfo> getRSGroup(String groupName) {
+ CompletableFuture<RSGroupInfo> future = new CompletableFuture<>();
+ addListener(listRSGroups(), (groups, err) -> {
+ if (err != null) {
+ future.completeExceptionally(err);
+ return;
+ }
+ for (RSGroupInfo rsGroupInfo : groups) {
+ if (rsGroupInfo.getName().equals(groupName)){
+ future.complete(rsGroupInfo);
+ return;
+ }
+ }
+ future.complete(null);
+ });
+ return future;
+ }
+
+
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java
index d45423c..55040cc 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java
@@ -26,7 +26,6 @@ import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
-
import org.apache.hadoop.hbase.CellScannable;
import org.apache.hadoop.hbase.ClusterMetrics.Option;
import org.apache.hadoop.hbase.ClusterMetricsBuilder;
@@ -56,6 +55,7 @@ import org.apache.hadoop.hbase.client.replication.ReplicationPeerConfigUtil;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.filter.ByteArrayComparable;
import org.apache.hadoop.hbase.io.TimeRange;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
import org.apache.hadoop.hbase.util.Bytes;
@@ -64,7 +64,9 @@ import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.security.token.Token;
import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
import org.apache.hbase.thirdparty.com.google.protobuf.UnsafeByteOperations;
+
import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.ClearCompactionQueuesRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.ClearRegionBlockCacheRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.CompactRegionRequest;
@@ -121,8 +123,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsCatalogJ
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsCleanerChoreEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsMasterRunningRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsNormalizerEnabledRequest;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .IsSnapshotCleanupEnabledRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotCleanupEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSplitOrMergeEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MergeTableRegionsRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ModifyColumnRequest;
@@ -137,8 +138,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.RunCleaner
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetBalancerRunningRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetCleanerChoreRunningRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetNormalizerRunningRequest;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .SetSnapshotCleanupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSnapshotCleanupRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSplitOrMergeEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetTableStateInMetaRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SplitTableRegionRequest;
@@ -147,6 +147,8 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.UnassignRe
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetQuotaStatesRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaRegionSizesRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaSnapshotsRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.GetLastFlushedSequenceIdRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationProtos;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationProtos.AddReplicationPeerRequest;
@@ -1934,4 +1936,29 @@ public final class RequestConverter {
return IsSnapshotCleanupEnabledRequest.newBuilder().build();
}
+ public static MoveServersRequest buildMoveServersRequest(Set<Address> servers,
+ String targetGroup) {
+ Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
+ for (Address el : servers) {
+ hostPorts.add(
+ HBaseProtos.ServerName.newBuilder().setHostName(el.getHostname()).setPort(el.getPort())
+ .build());
+ }
+ return MoveServersRequest.newBuilder().setTargetGroup(targetGroup).addAllServers(hostPorts)
+ .build();
+ }
+
+ public static RemoveServersRequest buildRemoveServersRequest(Set<Address> servers) {
+ Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
+ for(Address el: servers) {
+ hostPorts.add(HBaseProtos.ServerName.newBuilder()
+ .setHostName(el.getHostname())
+ .setPort(el.getPort())
+ .build());
+ }
+ return RemoveServersRequest.newBuilder()
+ .addAllServers(hostPorts)
+ .build();
+ }
+
}
diff --git a/hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java b/hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
index 4f8b3ad..ef3a93a 100644
--- a/hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
+++ b/hbase-it/src/test/java/org/apache/hadoop/hbase/rsgroup/IntegrationTestRSGroup.java
@@ -82,7 +82,7 @@ public class IntegrationTestRSGroup extends TestRSGroupsBase {
LOG.info("Waiting for cleanup to finish "+ rsGroupAdmin.listRSGroups());
//Might be greater since moving servers back to default
//is after starting a server
- return rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().size()
+ return rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().size()
>= NUM_SLAVES_BASE;
}
});
@@ -93,7 +93,7 @@ public class IntegrationTestRSGroup extends TestRSGroupsBase {
LOG.info("Waiting for regionservers to be registered "+ rsGroupAdmin.listRSGroups());
//Might be greater since moving servers back to default
//is after starting a server
- return rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().size()
+ return rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().size()
== getNumServers();
}
});
diff --git a/hbase-protocol-shaded/src/main/protobuf/Master.proto b/hbase-protocol-shaded/src/main/protobuf/Master.proto
index fee9ab8..37878f4 100644
--- a/hbase-protocol-shaded/src/main/protobuf/Master.proto
+++ b/hbase-protocol-shaded/src/main/protobuf/Master.proto
@@ -37,6 +37,7 @@ import "Quota.proto";
import "Replication.proto";
import "Snapshot.proto";
import "AccessControl.proto";
+import "RSGroupAdmin.proto";
/* Column-level protobufs */
@@ -1081,6 +1082,24 @@ service MasterService {
/** returns a list of namespace names */
rpc ListNamespaces(ListNamespacesRequest)
returns(ListNamespacesResponse);
+
+ rpc MoveServers(MoveServersRequest)
+ returns (MoveServersResponse);
+
+ rpc AddRSGroup(AddRSGroupRequest)
+ returns (AddRSGroupResponse);
+
+ rpc RemoveRSGroup(RemoveRSGroupRequest)
+ returns (RemoveRSGroupResponse);
+
+ rpc BalanceRSGroup(BalanceRSGroupRequest)
+ returns (BalanceRSGroupResponse);
+
+ rpc ListRSGroupInfos(ListRSGroupInfosRequest)
+ returns (ListRSGroupInfosResponse);
+
+ rpc RemoveServers(RemoveServersRequest)
+ returns (RemoveServersResponse);
}
// HBCK Service definitions.
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 8035953..0cb3578 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3840,7 +3840,7 @@ public class HMaster extends HRegionServer implements MasterServices {
public static void decorateMasterConfiguration(Configuration conf) {
String plugins = conf.get(HBASE_MASTER_LOGCLEANER_PLUGINS);
String cleanerClass = ReplicationLogCleaner.class.getCanonicalName();
- if (!plugins.contains(cleanerClass)) {
+ if (plugins == null || !plugins.contains(cleanerClass)) {
conf.set(HBASE_MASTER_LOGCLEANER_PLUGINS, plugins + "," + cleanerClass);
}
if (ReplicationUtils.isReplicationForBulkLoadDataEnabled(conf)) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
index 47ef3d0..abd297c 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
@@ -1772,7 +1772,8 @@ public class MasterCoprocessorHost
});
}
- public void preDecommissionRegionServers(List<ServerName> servers, boolean offload) throws IOException {
+ public void preDecommissionRegionServers(List<ServerName> servers, boolean offload)
+ throws IOException {
execOperation(coprocEnvironments.isEmpty() ? null : new MasterObserverOperation() {
@Override
public void call(MasterObserver observer) throws IOException {
@@ -1781,7 +1782,8 @@ public class MasterCoprocessorHost
});
}
- public void postDecommissionRegionServers(List<ServerName> servers, boolean offload) throws IOException {
+ public void postDecommissionRegionServers(List<ServerName> servers, boolean offload)
+ throws IOException {
execOperation(coprocEnvironments.isEmpty() ? null : new MasterObserverOperation() {
@Override
public void call(MasterObserver observer) throws IOException {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
index af408d6..e79663e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
@@ -94,6 +94,7 @@ import org.apache.hadoop.hbase.regionserver.RpcSchedulerFactory;
import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.Superusers;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.security.access.AccessChecker;
@@ -118,6 +119,7 @@ import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
import org.apache.hbase.thirdparty.com.google.protobuf.RpcController;
import org.apache.hbase.thirdparty.com.google.protobuf.ServiceException;
import org.apache.hbase.thirdparty.com.google.protobuf.UnsafeByteOperations;
@@ -303,6 +305,18 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetQuotaSta
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaRegionSizesRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaRegionSizesResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos.GetSpaceQuotaRegionSizesResponse.RegionSizes;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.AddRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.AddRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.MoveServersResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupAdminProtos.RemoveServersResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.FileArchiveNotificationRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.FileArchiveNotificationResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.GetLastFlushedSequenceIdRequest;
@@ -346,9 +360,9 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.Snapshot
*/
@InterfaceAudience.Private
@SuppressWarnings("deprecation")
-public class MasterRpcServices extends RSRpcServices
- implements MasterService.BlockingInterface, RegionServerStatusService.BlockingInterface,
- LockService.BlockingInterface, HbckService.BlockingInterface {
+public class MasterRpcServices extends RSRpcServices implements MasterService.BlockingInterface,
+ RegionServerStatusService.BlockingInterface,
+ LockService.BlockingInterface, HbckService.BlockingInterface {
private static final Logger LOG = LoggerFactory.getLogger(MasterRpcServices.class.getName());
private static final Logger AUDITLOG =
LoggerFactory.getLogger("SecurityLogger."+MasterRpcServices.class.getName());
@@ -2845,4 +2859,155 @@ public class MasterRpcServices extends RSRpcServices
}
return true;
}
+
+ @Override
+ public MoveServersResponse moveServers(RpcController controller, MoveServersRequest request)
+ throws ServiceException {
+ Set<Address> hostPorts = Sets.newHashSet();
+ MoveServersResponse.Builder builder = MoveServersResponse.newBuilder();
+ for (HBaseProtos.ServerName el : request.getServersList()) {
+ hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts + " to rsgroup " +
+ request.getTargetGroup());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preMoveServers(hostPorts, request.getTargetGroup());
+ }
+ master.getRSGroupInfoManager().moveServers(hostPorts, request.getTargetGroup());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postMoveServers(hostPorts, request.getTargetGroup());
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
+ @Override
+ public AddRSGroupResponse addRSGroup(RpcController controller, AddRSGroupRequest request)
+ throws ServiceException {
+ AddRSGroupResponse.Builder builder = AddRSGroupResponse.newBuilder();
+ LOG.info(master.getClientIdAuditPrefix() + " add rsgroup " + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
+ }
+ master.getRSGroupInfoManager().addRSGroup(new RSGroupInfo(request.getRSGroupName()));
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postAddRSGroup(request.getRSGroupName());
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
+ @Override
+ public RemoveRSGroupResponse removeRSGroup(RpcController controller, RemoveRSGroupRequest request)
+ throws ServiceException {
+ RemoveRSGroupResponse.Builder builder = RemoveRSGroupResponse.newBuilder();
+ LOG.info(master.getClientIdAuditPrefix() + " remove rsgroup " + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preRemoveRSGroup(request.getRSGroupName());
+ }
+ master.getRSGroupInfoManager().removeRSGroup(request.getRSGroupName());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postRemoveRSGroup(request.getRSGroupName());
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
+ @Override
+ public BalanceRSGroupResponse balanceRSGroup(RpcController controller,
+ BalanceRSGroupRequest request) throws ServiceException {
+ BalanceRSGroupResponse.Builder builder = BalanceRSGroupResponse.newBuilder();
+ LOG.info(
+ master.getClientIdAuditPrefix() + " balance rsgroup, group=" + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preBalanceRSGroup(request.getRSGroupName());
+ }
+ boolean balancerRan =
+ master.getRSGroupInfoManager().balanceRSGroup(request.getRSGroupName());
+ builder.setBalanceRan(balancerRan);
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postBalanceRSGroup(request.getRSGroupName(), balancerRan);
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
+ @Override
+ public ListRSGroupInfosResponse listRSGroupInfos(RpcController controller,
+ ListRSGroupInfosRequest request) throws ServiceException {
+ ListRSGroupInfosResponse.Builder builder = ListRSGroupInfosResponse.newBuilder();
+ LOG.info(master.getClientIdAuditPrefix() + " list rsgroup");
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preListRSGroups();
+ }
+ List<RSGroupInfo> rsGroupInfos = master.getRSGroupInfoManager().listRSGroups().stream()
+ .map(RSGroupInfo::new).collect(Collectors.toList());
+ Map<String, RSGroupInfo> name2Info = new HashMap<>();
+ List<TableDescriptor> needToFill =
+ new ArrayList<>(master.getTableDescriptors().getAll().values());
+ for (RSGroupInfo rsGroupInfo : rsGroupInfos) {
+ name2Info.put(rsGroupInfo.getName(), rsGroupInfo);
+ for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
+ if (rsGroupInfo.containsTable(td.getTableName())){
+ needToFill.remove(td);
+ }
+ }
+ }
+ for (TableDescriptor td : needToFill) {
+ String groupName = td.getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo rsGroupInfo = name2Info.get(groupName);
+ if (rsGroupInfo != null) {
+ rsGroupInfo.addTable(td.getTableName());
+ }
+ }
+ for (RSGroupInfo rsGroupInfo : rsGroupInfos) {
+ // TODO: this can be done at once outside this loop, do not need to scan all every time.
+ builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postListRSGroups();
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
+ @Override
+ public RemoveServersResponse removeServers(RpcController controller,
+ RemoveServersRequest request) throws ServiceException {
+ RemoveServersResponse.Builder builder = RemoveServersResponse.newBuilder();
+ Set<Address> servers = Sets.newHashSet();
+ for (HBaseProtos.ServerName el : request.getServersList()) {
+ servers.add(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ LOG.info(master.getClientIdAuditPrefix() + " remove decommissioned servers from rsgroup: " +
+ servers);
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preRemoveServers(servers);
+ }
+ master.getRSGroupInfoManager().removeServers(servers);
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postRemoveServers(servers);
+ }
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ return builder.build();
+ }
+
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
index ae55352..68e5c76 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
@@ -52,6 +52,7 @@ import org.apache.hadoop.hbase.replication.SyncReplicationState;
import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.hadoop.hbase.security.access.ZKPermissionWatcher;
+import org.apache.hadoop.hbase.zookeeper.LoadBalancerTracker;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
@@ -540,4 +541,12 @@ public interface MasterServices extends Server {
* @return the {@link RSGroupInfoManager}
*/
RSGroupInfoManager getRSGroupInfoManager();
+
+ /**
+ * Queries the state of the {@link LoadBalancerTracker}. If the balancer is not initialized,
+ * false is returned.
+ *
+ * @return The state of the load balancer, or false if the load balancer isn't defined.
+ */
+ boolean isBalancerOn();
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java
index c7c521c..30dd4d7 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java
@@ -57,8 +57,7 @@ class DisabledRSGroupInfoManager implements RSGroupInfoManager {
}
@Override
- public Set<Address> moveServers(Set<Address> servers, String srcGroup, String dstGroup)
- throws IOException {
+ public void moveServers(Set<Address> servers, String targetGroupName) throws IOException {
throw new DoNotRetryIOException("RSGroup is disabled");
}
@@ -107,4 +106,14 @@ class DisabledRSGroupInfoManager implements RSGroupInfoManager {
public RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException {
return null;
}
+
+ @Override
+ public boolean balanceRSGroup(String groupName) throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
+
+ @Override
+ public void setRSGroup(Set<TableName> tables, String groupName) throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
index 344d0b3..3de6965 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
@@ -20,12 +20,16 @@ package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
import java.util.List;
import java.util.Set;
+import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
/**
* Group user API interface used between client and server.
+ *
+ * @deprecated Keep it here only for tests, using {@link Admin} instead.
*/
+@Deprecated
@InterfaceAudience.Private
public interface RSGroupAdmin {
/**
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
index 07f0efd..7cf4ed1 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
@@ -20,12 +20,33 @@ package org.apache.hadoop.hbase.rsgroup;
import com.google.protobuf.ServiceException;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.EnumSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.regex.Pattern;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.CacheEvictionStats;
+import org.apache.hadoop.hbase.ClusterMetrics;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.NamespaceNotFoundException;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableExistsException;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactType;
+import org.apache.hadoop.hbase.client.CompactionState;
import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.SnapshotDescription;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.replication.TableCFs;
+import org.apache.hadoop.hbase.client.security.SecurityCapability;
+import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
@@ -38,13 +59,26 @@ import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupI
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoResponse;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RSGroupAdminService;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
+import org.apache.hadoop.hbase.quotas.QuotaFilter;
+import org.apache.hadoop.hbase.quotas.QuotaSettings;
+import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshotView;
+import org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
+import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
+import org.apache.hadoop.hbase.security.access.Permission;
+import org.apache.hadoop.hbase.security.access.UserPermission;
+import org.apache.hadoop.hbase.snapshot.HBaseSnapshotException;
+import org.apache.hadoop.hbase.snapshot.RestoreSnapshotException;
+import org.apache.hadoop.hbase.snapshot.SnapshotCreationException;
+import org.apache.hadoop.hbase.snapshot.UnknownSnapshotException;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
@@ -52,27 +86,866 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
/**
* Client used for managing region server group information.
+ *
+ * @deprecated Keep it here only for tests, using {@link Admin} instead.
*/
+@Deprecated
@InterfaceAudience.Private
-public class RSGroupAdminClient implements RSGroupAdmin {
+public class RSGroupAdminClient implements RSGroupAdmin, Admin {
private RSGroupAdminService.BlockingInterface stub;
private Admin admin;
- public RSGroupAdminClient(Connection conn) throws IOException {
- admin = conn.getAdmin();
- stub = RSGroupAdminService.newBlockingStub(admin.coprocessorService());
+ public RSGroupAdminClient(Connection conn) throws IOException {
+ admin = conn.getAdmin();
+ stub = RSGroupAdminService.newBlockingStub(admin.coprocessorService());
+ }
+
+ // for writing UTs
+ @VisibleForTesting
+ protected RSGroupAdminClient() {
+ }
+
+ @Override
+ public int getOperationTimeout() {
+ return 0;
+ }
+
+ @Override
+ public int getSyncWaitTimeout() {
+ return 0;
+ }
+
+ @Override
+ public void abort(String why, Throwable e) {
+
+ }
+
+ @Override
+ public boolean isAborted() {
+ return false;
+ }
+
+ @Override
+ public Connection getConnection() {
+ return null;
+ }
+
+ @Override
+ public boolean tableExists(TableName tableName) throws IOException {
+ return false;
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptors() throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptors(boolean includeSysTables) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptors(Pattern pattern) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptors(Pattern pattern, boolean includeSysTables)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public TableName[] listTableNames() throws IOException {
+ return new TableName[0];
+ }
+
+ @Override
+ public TableName[] listTableNames(Pattern pattern, boolean includeSysTables) throws IOException {
+ return new TableName[0];
+ }
+
+ @Override
+ public TableDescriptor getDescriptor(TableName tableName)
+ throws TableNotFoundException, IOException {
+ return null;
+ }
+
+ @Override
+ public void createTable(TableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions)
+ throws IOException {
+
+ }
+
+ @Override
+ public Future<Void> createTableAsync(TableDescriptor desc) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> createTableAsync(TableDescriptor desc, byte[][] splitKeys)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> deleteTableAsync(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> truncateTableAsync(TableName tableName, boolean preserveSplits)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> enableTableAsync(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> disableTableAsync(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public boolean isTableEnabled(TableName tableName) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isTableDisabled(TableName tableName) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isTableAvailable(TableName tableName) throws IOException {
+ return false;
+ }
+
+ @Override
+ public Future<Void> addColumnFamilyAsync(TableName tableName, ColumnFamilyDescriptor columnFamily)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> deleteColumnFamilyAsync(TableName tableName, byte[] columnFamily)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> modifyColumnFamilyAsync(TableName tableName,
+ ColumnFamilyDescriptor columnFamily) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<RegionInfo> getRegions(ServerName serverName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void flush(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void flushRegion(byte[] regionName) throws IOException {
+
+ }
+
+ @Override
+ public void flushRegionServer(ServerName serverName) throws IOException {
+
+ }
+
+ @Override
+ public void compact(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void compactRegion(byte[] regionName) throws IOException {
+
+ }
+
+ @Override
+ public void compact(TableName tableName, byte[] columnFamily) throws IOException {
+
+ }
+
+ @Override
+ public void compactRegion(byte[] regionName, byte[] columnFamily) throws IOException {
+
+ }
+
+ @Override
+ public void compact(TableName tableName, CompactType compactType)
+ throws IOException, InterruptedException {
+
+ }
+
+ @Override
+ public void compact(TableName tableName, byte[] columnFamily, CompactType compactType)
+ throws IOException, InterruptedException {
+
+ }
+
+ @Override
+ public void majorCompact(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void majorCompactRegion(byte[] regionName) throws IOException {
+
+ }
+
+ @Override
+ public void majorCompact(TableName tableName, byte[] columnFamily) throws IOException {
+
+ }
+
+ @Override
+ public void majorCompactRegion(byte[] regionName, byte[] columnFamily) throws IOException {
+
+ }
+
+ @Override
+ public void majorCompact(TableName tableName, CompactType compactType)
+ throws IOException, InterruptedException {
+
+ }
+
+ @Override
+ public void majorCompact(TableName tableName, byte[] columnFamily, CompactType compactType)
+ throws IOException, InterruptedException {
+
+ }
+
+ @Override
+ public Map<ServerName, Boolean> compactionSwitch(boolean switchState,
+ List<String> serverNamesList) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void compactRegionServer(ServerName serverName) throws IOException {
+
+ }
+
+ @Override
+ public void majorCompactRegionServer(ServerName serverName) throws IOException {
+
+ }
+
+ @Override
+ public void move(byte[] encodedRegionName) throws IOException {
+
+ }
+
+ @Override
+ public void move(byte[] encodedRegionName, ServerName destServerName) throws IOException {
+
+ }
+
+ @Override
+ public void assign(byte[] regionName) throws IOException {
+
+ }
+
+ @Override
+ public void unassign(byte[] regionName, boolean force) throws IOException {
+
+ }
+
+ @Override
+ public void offline(byte[] regionName) throws IOException {
+
+ }
+
+ @Override
+ public boolean balancerSwitch(boolean onOrOff, boolean synchronous) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean balance() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean balance(boolean force) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isBalancerEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public CacheEvictionStats clearBlockCache(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public boolean normalize() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isNormalizerEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean normalizerSwitch(boolean on) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean catalogJanitorSwitch(boolean onOrOff) throws IOException {
+ return false;
+ }
+
+ @Override
+ public int runCatalogJanitor() throws IOException {
+ return 0;
+ }
+
+ @Override
+ public boolean isCatalogJanitorEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean cleanerChoreSwitch(boolean onOrOff) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean runCleanerChore() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isCleanerChoreEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public Future<Void> mergeRegionsAsync(byte[][] nameofRegionsToMerge, boolean forcible)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public void split(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void split(TableName tableName, byte[] splitPoint) throws IOException {
+
+ }
+
+ @Override
+ public Future<Void> splitRegionAsync(byte[] regionName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> splitRegionAsync(byte[] regionName, byte[] splitPoint) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> modifyTableAsync(TableDescriptor td) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void shutdown() throws IOException {
+
+ }
+
+ @Override
+ public void stopMaster() throws IOException {
+
+ }
+
+ @Override
+ public boolean isMasterInMaintenanceMode() throws IOException {
+ return false;
+ }
+
+ @Override
+ public void stopRegionServer(String hostnamePort) throws IOException {
+
+ }
+
+ @Override
+ public ClusterMetrics getClusterMetrics(EnumSet<ClusterMetrics.Option> options)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<RegionMetrics> getRegionMetrics(ServerName serverName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<RegionMetrics> getRegionMetrics(ServerName serverName, TableName tableName)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public Configuration getConfiguration() {
+ return null;
+ }
+
+ @Override
+ public Future<Void> createNamespaceAsync(NamespaceDescriptor descriptor) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> modifyNamespaceAsync(NamespaceDescriptor descriptor) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> deleteNamespaceAsync(String name) throws IOException {
+ return null;
+ }
+
+ @Override
+ public NamespaceDescriptor getNamespaceDescriptor(String name)
+ throws NamespaceNotFoundException, IOException {
+ return null;
+ }
+
+ @Override
+ public String[] listNamespaces() throws IOException {
+ return new String[0];
+ }
+
+ @Override
+ public NamespaceDescriptor[] listNamespaceDescriptors() throws IOException {
+ return new NamespaceDescriptor[0];
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptorsByNamespace(byte[] name) throws IOException {
+ return null;
+ }
+
+ @Override
+ public TableName[] listTableNamesByNamespace(String name) throws IOException {
+ return new TableName[0];
+ }
+
+ @Override
+ public List<RegionInfo> getRegions(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void close() {
+
+ }
+
+ @Override
+ public List<TableDescriptor> listTableDescriptors(List<TableName> tableNames) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Boolean> abortProcedureAsync(long procId, boolean mayInterruptIfRunning)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public String getProcedures() throws IOException {
+ return null;
+ }
+
+ @Override
+ public String getLocks() throws IOException {
+ return null;
+ }
+
+ @Override
+ public void rollWALWriter(ServerName serverName) throws IOException, FailedLogCloseException {
+
+ }
+
+ @Override
+ public CompactionState getCompactionState(TableName tableName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public CompactionState getCompactionState(TableName tableName, CompactType compactType)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public CompactionState getCompactionStateForRegion(byte[] regionName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public long getLastMajorCompactionTimestamp(TableName tableName) throws IOException {
+ return 0;
+ }
+
+ @Override
+ public long getLastMajorCompactionTimestampForRegion(byte[] regionName) throws IOException {
+ return 0;
+ }
+
+ @Override
+ public void snapshot(SnapshotDescription snapshot)
+ throws IOException, SnapshotCreationException, IllegalArgumentException {
+
+ }
+
+ @Override
+ public Future<Void> snapshotAsync(SnapshotDescription snapshot)
+ throws IOException, SnapshotCreationException {
+ return null;
+ }
+
+ @Override
+ public boolean isSnapshotFinished(SnapshotDescription snapshot)
+ throws IOException, HBaseSnapshotException, UnknownSnapshotException {
+ return false;
+ }
+
+ @Override
+ public void restoreSnapshot(String snapshotName) throws IOException, RestoreSnapshotException {
+
+ }
+
+ @Override
+ public void restoreSnapshot(String snapshotName, boolean takeFailSafeSnapshot, boolean restoreAcl)
+ throws IOException, RestoreSnapshotException {
+
+ }
+
+ @Override
+ public Future<Void> cloneSnapshotAsync(String snapshotName, TableName tableName,
+ boolean restoreAcl) throws IOException, TableExistsException, RestoreSnapshotException {
+ return null;
+ }
+
+ @Override
+ public void execProcedure(String signature, String instance, Map<String, String> props)
+ throws IOException {
+
+ }
+
+ @Override
+ public byte[] execProcedureWithReturn(String signature, String instance,
+ Map<String, String> props) throws IOException {
+ return new byte[0];
+ }
+
+ @Override
+ public boolean isProcedureFinished(String signature, String instance, Map<String, String> props)
+ throws IOException {
+ return false;
+ }
+
+ @Override
+ public List<SnapshotDescription> listSnapshots() throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<SnapshotDescription> listSnapshots(Pattern pattern) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<SnapshotDescription> listTableSnapshots(Pattern tableNamePattern,
+ Pattern snapshotNamePattern) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void deleteSnapshot(String snapshotName) throws IOException {
+
}
- // for writing UTs
- @VisibleForTesting
- protected RSGroupAdminClient() {
+ @Override
+ public void deleteSnapshots(Pattern pattern) throws IOException {
+
+ }
+
+ @Override
+ public void deleteTableSnapshots(Pattern tableNamePattern, Pattern snapshotNamePattern)
+ throws IOException {
+
+ }
+
+ @Override
+ public void setQuota(QuotaSettings quota) throws IOException {
+
+ }
+
+ @Override
+ public List<QuotaSettings> getQuota(QuotaFilter filter) throws IOException {
+ return null;
+ }
+
+ @Override
+ public CoprocessorRpcChannel coprocessorService() {
+ return null;
+ }
+
+ @Override
+ public CoprocessorRpcChannel coprocessorService(ServerName serverName) {
+ return null;
+ }
+
+ @Override
+ public void updateConfiguration(ServerName server) throws IOException {
+
+ }
+
+ @Override
+ public void updateConfiguration() throws IOException {
+
+ }
+
+ @Override
+ public List<SecurityCapability> getSecurityCapabilities() throws IOException {
+ return null;
+ }
+
+ @Override
+ public boolean splitSwitch(boolean enabled, boolean synchronous) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean mergeSwitch(boolean enabled, boolean synchronous) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isSplitEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isMergeEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public Future<Void> addReplicationPeerAsync(String peerId, ReplicationPeerConfig peerConfig,
+ boolean enabled) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> removeReplicationPeerAsync(String peerId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> enableReplicationPeerAsync(String peerId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> disableReplicationPeerAsync(String peerId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public ReplicationPeerConfig getReplicationPeerConfig(String peerId) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> updateReplicationPeerConfigAsync(String peerId,
+ ReplicationPeerConfig peerConfig) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<ReplicationPeerDescription> listReplicationPeers() throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<ReplicationPeerDescription> listReplicationPeers(Pattern pattern) throws IOException {
+ return null;
+ }
+
+ @Override
+ public Future<Void> transitReplicationPeerSyncReplicationStateAsync(String peerId,
+ SyncReplicationState state) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void decommissionRegionServers(List<ServerName> servers, boolean offload)
+ throws IOException {
+
+ }
+
+ @Override
+ public List<ServerName> listDecommissionedRegionServers() throws IOException {
+ return null;
+ }
+
+ @Override
+ public void recommissionRegionServer(ServerName server, List<byte[]> encodedRegionNames)
+ throws IOException {
+
+ }
+
+ @Override
+ public List<TableCFs> listReplicatedTableCFs() throws IOException {
+ return null;
+ }
+
+ @Override
+ public void enableTableReplication(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void disableTableReplication(TableName tableName) throws IOException {
+
+ }
+
+ @Override
+ public void clearCompactionQueues(ServerName serverName, Set<String> queues)
+ throws IOException, InterruptedException {
+
+ }
+
+ @Override
+ public List<ServerName> clearDeadServers(List<ServerName> servers) throws IOException {
+ return null;
+ }
+
+ @Override
+ public void cloneTableSchema(TableName tableName, TableName newTableName, boolean preserveSplits)
+ throws IOException {
+
+ }
+
+ @Override
+ public boolean switchRpcThrottle(boolean enable) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isRpcThrottleEnabled() throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean exceedThrottleQuotaSwitch(boolean enable) throws IOException {
+ return false;
+ }
+
+ @Override
+ public Map<TableName, Long> getSpaceQuotaTableSizes() throws IOException {
+ return null;
+ }
+
+ @Override
+ public Map<TableName, ? extends SpaceQuotaSnapshotView> getRegionServerSpaceQuotaSnapshots(
+ ServerName serverName) throws IOException {
+ return null;
+ }
+
+ @Override
+ public SpaceQuotaSnapshotView getCurrentSpaceQuotaSnapshot(String namespace) throws IOException {
+ return null;
+ }
+
+ @Override
+ public SpaceQuotaSnapshotView getCurrentSpaceQuotaSnapshot(TableName tableName)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public void grant(UserPermission userPermission, boolean mergeExistingPermissions)
+ throws IOException {
+
+ }
+
+ @Override
+ public void revoke(UserPermission userPermission) throws IOException {
+
+ }
+
+ @Override
+ public List<UserPermission> getUserPermissions(
+ GetUserPermissionsRequest getUserPermissionsRequest) throws IOException {
+ return null;
+ }
+
+ @Override
+ public List<Boolean> hasUserPermissions(String userName, List<Permission> permissions)
+ throws IOException {
+ return null;
+ }
+
+ @Override
+ public boolean snapshotCleanupSwitch(boolean on, boolean synchronous) throws IOException {
+ return false;
+ }
+
+ @Override
+ public boolean isSnapshotCleanupEnabled() throws IOException {
+ return false;
}
@Override
public RSGroupInfo getRSGroupInfo(String groupName) throws IOException {
+ return getRSGroup(groupName);
+ }
+
+ @Override
+ public void moveServers(Set<Address> servers, String targetGroup) throws IOException {
+ moveToRSGroup(servers, targetGroup);
+ }
+
+ @Override
+ public void addRSGroup(String groupName) throws IOException {
+ AddRSGroupRequest request = AddRSGroupRequest.newBuilder().setRSGroupName(groupName).build();
+ try {
+ stub.addRSGroup(null, request);
+ } catch (ServiceException e) {
+ throw ProtobufUtil.handleRemoteException(e);
+ }
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(String groupName) throws IOException {
try {
GetRSGroupInfoResponse resp = stub.getRSGroupInfo(null,
- GetRSGroupInfoRequest.newBuilder().setRSGroupName(groupName).build());
+ GetRSGroupInfoRequest.newBuilder().setRSGroupName(groupName).build());
if (resp.hasRSGroupInfo()) {
return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
@@ -82,11 +955,13 @@ public class RSGroupAdminClient implements RSGroupAdmin {
}
}
- public RSGroupInfo getRSGroupInfoOfTable(TableName tableName) throws IOException {
- GetRSGroupInfoOfTableRequest request = GetRSGroupInfoOfTableRequest.newBuilder().setTableName(
- ProtobufUtil.toProtoTableName(tableName)).build();
+ @Override
+ public RSGroupInfo getRSGroup(Address hostPort) throws IOException {
+ GetRSGroupInfoOfServerRequest request = GetRSGroupInfoOfServerRequest.newBuilder().setServer(
+ HBaseProtos.ServerName.newBuilder().setHostName(hostPort.getHostname())
+ .setPort(hostPort.getPort()).build()).build();
try {
- GetRSGroupInfoOfTableResponse resp = stub.getRSGroupInfoOfTable(null, request);
+ GetRSGroupInfoOfServerResponse resp = stub.getRSGroupInfoOfServer(null, request);
if (resp.hasRSGroupInfo()) {
return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
@@ -97,55 +972,76 @@ public class RSGroupAdminClient implements RSGroupAdmin {
}
@Override
- public void moveServers(Set<Address> servers, String targetGroup) throws IOException {
- Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
- for(Address el: servers) {
- hostPorts.add(HBaseProtos.ServerName.newBuilder()
- .setHostName(el.getHostname())
- .setPort(el.getPort())
- .build());
+ public RSGroupInfo getRSGroup(TableName tableName) throws IOException {
+ GetRSGroupInfoOfTableRequest request = GetRSGroupInfoOfTableRequest.newBuilder()
+ .setTableName(ProtobufUtil.toProtoTableName(tableName)).build();
+ try {
+ GetRSGroupInfoOfTableResponse resp = stub.getRSGroupInfoOfTable(null, request);
+ if (resp.hasRSGroupInfo()) {
+ return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
+ }
+ return null;
+ } catch (ServiceException e) {
+ throw ProtobufUtil.handleRemoteException(e);
}
- MoveServersRequest request = MoveServersRequest.newBuilder()
- .setTargetGroup(targetGroup)
- .addAllServers(hostPorts)
- .build();
+ }
+
+ @Override
+ public void removeRSGroup(String name) throws IOException {
+ RemoveRSGroupRequest request = RemoveRSGroupRequest.newBuilder().setRSGroupName(name).build();
try {
- stub.moveServers(null, request);
+ stub.removeRSGroup(null, request);
} catch (ServiceException e) {
throw ProtobufUtil.handleRemoteException(e);
}
}
- public void moveTables(Set<TableName> tables, String targetGroup) throws IOException {
- MoveTablesRequest.Builder builder = MoveTablesRequest.newBuilder().setTargetGroup(targetGroup);
- for(TableName tableName: tables) {
- builder.addTableName(ProtobufUtil.toProtoTableName(tableName));
- if (!admin.tableExists(tableName)) {
- throw new TableNotFoundException(tableName);
- }
+ @Override
+ public void removeRSGroup(Set<Address> servers) throws IOException {
+ Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
+ for (Address el : servers) {
+ hostPorts.add(
+ HBaseProtos.ServerName.newBuilder().setHostName(el.getHostname()).setPort(el.getPort())
+ .build());
}
+ RemoveServersRequest request =
+ RemoveServersRequest.newBuilder().addAllServers(hostPorts).build();
try {
- stub.moveTables(null, builder.build());
+ stub.removeServers(null, request);
} catch (ServiceException e) {
throw ProtobufUtil.handleRemoteException(e);
}
}
@Override
- public void addRSGroup(String groupName) throws IOException {
- AddRSGroupRequest request = AddRSGroupRequest.newBuilder().setRSGroupName(groupName).build();
+ public void moveToRSGroup(Set<Address> servers, String targetGroup) throws IOException {
+ Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
+ for (Address el : servers) {
+ hostPorts.add(
+ HBaseProtos.ServerName.newBuilder().setHostName(el.getHostname()).setPort(el.getPort())
+ .build());
+ }
+ MoveServersRequest request =
+ MoveServersRequest.newBuilder().setTargetGroup(targetGroup).addAllServers(hostPorts)
+ .build();
try {
- stub.addRSGroup(null, request);
+ stub.moveServers(null, request);
} catch (ServiceException e) {
throw ProtobufUtil.handleRemoteException(e);
}
}
@Override
- public void removeRSGroup(String name) throws IOException {
- RemoveRSGroupRequest request = RemoveRSGroupRequest.newBuilder().setRSGroupName(name).build();
+ public void setRSGroup(Set<TableName> tables, String groupName) throws IOException {
+ MoveTablesRequest.Builder builder = MoveTablesRequest.newBuilder().setTargetGroup(groupName);
+ for (TableName tableName : tables) {
+ builder.addTableName(ProtobufUtil.toProtoTableName(tableName));
+ if (!admin.tableExists(tableName)) {
+ throw new TableNotFoundException(tableName);
+ }
+ }
try {
- stub.removeRSGroup(null, request);
+ stub.moveTables(null, builder.build());
} catch (ServiceException e) {
throw ProtobufUtil.handleRemoteException(e);
}
@@ -153,8 +1049,8 @@ public class RSGroupAdminClient implements RSGroupAdmin {
@Override
public boolean balanceRSGroup(String groupName) throws IOException {
- BalanceRSGroupRequest request = BalanceRSGroupRequest.newBuilder()
- .setRSGroupName(groupName).build();
+ BalanceRSGroupRequest request =
+ BalanceRSGroupRequest.newBuilder().setRSGroupName(groupName).build();
try {
return stub.balanceRSGroup(null, request).getBalanceRan();
} catch (ServiceException e) {
@@ -165,10 +1061,11 @@ public class RSGroupAdminClient implements RSGroupAdmin {
@Override
public List<RSGroupInfo> listRSGroups() throws IOException {
try {
- List<RSGroupProtos.RSGroupInfo> resp = stub.listRSGroupInfos(null,
- ListRSGroupInfosRequest.getDefaultInstance()).getRSGroupInfoList();
+ List<RSGroupProtos.RSGroupInfo> resp =
+ stub.listRSGroupInfos(null, ListRSGroupInfosRequest.getDefaultInstance())
+ .getRSGroupInfoList();
List<RSGroupInfo> result = new ArrayList<>(resp.size());
- for(RSGroupProtos.RSGroupInfo entry : resp) {
+ for (RSGroupProtos.RSGroupInfo entry : resp) {
result.add(ProtobufUtil.toGroupInfo(entry));
}
return result;
@@ -179,62 +1076,12 @@ public class RSGroupAdminClient implements RSGroupAdmin {
@Override
public RSGroupInfo getRSGroupOfServer(Address hostPort) throws IOException {
- GetRSGroupInfoOfServerRequest request = GetRSGroupInfoOfServerRequest.newBuilder()
- .setServer(HBaseProtos.ServerName.newBuilder()
- .setHostName(hostPort.getHostname())
- .setPort(hostPort.getPort())
- .build())
- .build();
- try {
- GetRSGroupInfoOfServerResponse resp = stub.getRSGroupInfoOfServer(null, request);
- if (resp.hasRSGroupInfo()) {
- return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
- }
- return null;
- } catch (ServiceException e) {
- throw ProtobufUtil.handleRemoteException(e);
- }
- }
-
- public void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String targetGroup)
- throws IOException {
- MoveServersAndTablesRequest.Builder builder =
- MoveServersAndTablesRequest.newBuilder().setTargetGroup(targetGroup);
- for(Address el: servers) {
- builder.addServers(HBaseProtos.ServerName.newBuilder()
- .setHostName(el.getHostname())
- .setPort(el.getPort())
- .build());
- }
- for(TableName tableName: tables) {
- builder.addTableName(ProtobufUtil.toProtoTableName(tableName));
- if (!admin.tableExists(tableName)) {
- throw new TableNotFoundException(tableName);
- }
- }
- try {
- stub.moveServersAndTables(null, builder.build());
- } catch (ServiceException e) {
- throw ProtobufUtil.handleRemoteException(e);
- }
+ return getRSGroup(hostPort);
}
@Override
public void removeServers(Set<Address> servers) throws IOException {
- Set<HBaseProtos.ServerName> hostPorts = Sets.newHashSet();
- for(Address el: servers) {
- hostPorts.add(HBaseProtos.ServerName.newBuilder()
- .setHostName(el.getHostname())
- .setPort(el.getPort())
- .build());
- }
- RemoveServersRequest request = RemoveServersRequest.newBuilder()
- .addAllServers(hostPorts)
- .build();
- try {
- stub.removeServers(null, request);
- } catch (ServiceException e) {
- throw ProtobufUtil.handleRemoteException(e);
- }
+ removeRSGroup(servers);
}
+
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 353b4d2..621e822 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -25,12 +25,8 @@ import org.apache.hadoop.hbase.coprocessor.CoreCoprocessor;
import org.apache.hadoop.hbase.coprocessor.HasMasterServices;
import org.apache.hadoop.hbase.coprocessor.MasterCoprocessor;
import org.apache.hadoop.hbase.master.MasterServices;
-import org.apache.hadoop.hbase.security.UserProvider;
-import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.yetus.audience.InterfaceAudience;
-import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
-
/**
* @deprecated Keep it here only for compatibility with old client, all the logics have been moved
* into core of HBase.
@@ -42,8 +38,6 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor {
// Only instance of RSGroupInfoManager. RSGroup aware load balancers ask for this instance on
// their setup.
private MasterServices master;
- private RSGroupInfoManager groupInfoManager;
- private RSGroupAdminServer groupAdminServer;
private RSGroupAdminServiceImpl groupAdminService = new RSGroupAdminServiceImpl();
@Override
@@ -51,19 +45,8 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor {
if (!(env instanceof HasMasterServices)) {
throw new IOException("Does not implement HMasterServices");
}
-
master = ((HasMasterServices) env).getMasterServices();
- groupInfoManager = master.getRSGroupInfoManager();
- groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
- AccessChecker accessChecker = ((HasMasterServices) env).getMasterServices().getAccessChecker();
-
- // set the user-provider.
- UserProvider userProvider = UserProvider.instantiate(env.getConfiguration());
- groupAdminService.initialize(master, groupAdminServer, accessChecker, userProvider);
- }
-
- @Override
- public void stop(CoprocessorEnvironment env) {
+ groupAdminService.initialize(master);
}
@Override
@@ -72,11 +55,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor {
}
RSGroupInfoManager getGroupInfoManager() {
- return groupInfoManager;
+ return master.getRSGroupInfoManager();
}
- @VisibleForTesting
- RSGroupAdminServiceImpl getGroupAdminService() {
- return groupAdminService;
- }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
deleted file mode 100644
index 1e324e5..0000000
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
+++ /dev/null
@@ -1,495 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hbase.rsgroup;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-import java.util.Set;
-import java.util.function.Function;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.hbase.DoNotRetryIOException;
-import org.apache.hadoop.hbase.NamespaceDescriptor;
-import org.apache.hadoop.hbase.ServerName;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.RegionInfo;
-import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.constraint.ConstraintException;
-import org.apache.hadoop.hbase.master.HMaster;
-import org.apache.hadoop.hbase.master.LoadBalancer;
-import org.apache.hadoop.hbase.master.MasterServices;
-import org.apache.hadoop.hbase.master.RegionPlan;
-import org.apache.hadoop.hbase.master.RegionState;
-import org.apache.hadoop.hbase.master.ServerManager;
-import org.apache.hadoop.hbase.master.assignment.RegionStateNode;
-import org.apache.hadoop.hbase.net.Address;
-import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
-
-/**
- * Service to support Region Server Grouping (HBase-6721).
- */
-@InterfaceAudience.Private
-public class RSGroupAdminServer implements RSGroupAdmin {
- private static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminServer.class);
- static final String KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE = "should keep at least " +
- "one server in 'default' RSGroup.";
-
- private MasterServices master;
- final RSGroupInfoManager rsGroupInfoManager;
-
- /** Define the config key of retries threshold when movements failed */
- //made package private for testing
- static final String FAILED_MOVE_MAX_RETRY = "hbase.rsgroup.move.max.retry";
-
- /** Define the default number of retries */
- //made package private for testing
- static final int DEFAULT_MAX_RETRY_VALUE = 50;
-
- private int moveMaxRetry;
-
- public RSGroupAdminServer(MasterServices master, RSGroupInfoManager rsGroupInfoManager) {
- this.master = master;
- this.rsGroupInfoManager = rsGroupInfoManager;
- this.moveMaxRetry = master.getConfiguration().getInt(FAILED_MOVE_MAX_RETRY,
- DEFAULT_MAX_RETRY_VALUE);
- }
-
- @Override
- public RSGroupInfo getRSGroupInfo(String groupName) throws IOException {
- return rsGroupInfoManager.getRSGroup(groupName);
- }
-
- private void checkOnlineServersOnly(Set<Address> servers) throws ConstraintException {
- // This uglyness is because we only have Address, not ServerName.
- // Online servers are keyed by ServerName.
- Set<Address> onlineServers = new HashSet<>();
- for(ServerName server: master.getServerManager().getOnlineServers().keySet()) {
- onlineServers.add(server.getAddress());
- }
- for (Address address: servers) {
- if (!onlineServers.contains(address)) {
- throw new ConstraintException(
- "Server " + address + " is not an online server in 'default' RSGroup.");
- }
- }
- }
-
- /**
- * Check passed name. Fail if nulls or if corresponding RSGroupInfo not found.
- * @return The RSGroupInfo named <code>name</code>
- */
- private RSGroupInfo getAndCheckRSGroupInfo(String name) throws IOException {
- if (StringUtils.isEmpty(name)) {
- throw new ConstraintException("RSGroup cannot be null.");
- }
- RSGroupInfo rsGroupInfo = getRSGroupInfo(name);
- if (rsGroupInfo == null) {
- throw new ConstraintException("RSGroup does not exist: " + name);
- }
- return rsGroupInfo;
- }
-
- /**
- * @return List of Regions associated with this <code>server</code>.
- */
- private List<RegionInfo> getRegions(final Address server) {
- LinkedList<RegionInfo> regions = new LinkedList<>();
- for (Map.Entry<RegionInfo, ServerName> el :
- master.getAssignmentManager().getRegionStates().getRegionAssignments().entrySet()) {
- if (el.getValue() == null) {
- continue;
- }
-
- if (el.getValue().getAddress().equals(server)) {
- addRegion(regions, el.getKey());
- }
- }
- for (RegionStateNode state : master.getAssignmentManager().getRegionsInTransition()) {
- if (state.getRegionLocation() != null &&
- state.getRegionLocation().getAddress().equals(server)) {
- addRegion(regions, state.getRegionInfo());
- }
- }
- return regions;
- }
-
- private void addRegion(final LinkedList<RegionInfo> regions, RegionInfo hri) {
- // If meta, move it last otherwise other unassigns fail because meta is not
- // online for them to update state in. This is dodgy. Needs to be made more
- // robust. See TODO below.
- if (hri.isMetaRegion()) {
- regions.addLast(hri);
- } else {
- regions.addFirst(hri);
- }
- }
-
- /**
- * Move every region from servers which are currently located on these servers, but should not be
- * located there.
- * @param servers the servers that will move to new group
- * @param targetGroupName the target group name
- * @throws IOException if moving the server and tables fail
- */
- private void moveServerRegionsFromGroup(Set<Address> servers, String targetGroupName)
- throws IOException {
- moveRegionsBetweenGroups(servers, targetGroupName, rs -> getRegions(rs), info -> {
- try {
- String groupName = RSGroupUtil.getRSGroupInfo(master, rsGroupInfoManager, info.getTable())
- .map(RSGroupInfo::getName).orElse(RSGroupInfo.DEFAULT_GROUP);
- return groupName.equals(targetGroupName);
- } catch (IOException e) {
- LOG.warn("Failed to test group for region {} and target group {}", info, targetGroupName);
- return false;
- }
- }, rs -> rs.getHostname());
- }
-
- private <T> void moveRegionsBetweenGroups(Set<T> regionsOwners, String targetGroupName,
- Function<T, List<RegionInfo>> getRegionsInfo, Function<RegionInfo, Boolean> validation,
- Function<T, String> getOwnerName) throws IOException {
- boolean hasRegionsToMove;
- int retry = 0;
- Set<T> allOwners = new HashSet<>(regionsOwners);
- Set<String> failedRegions = new HashSet<>();
- IOException toThrow = null;
- do {
- hasRegionsToMove = false;
- for (Iterator<T> iter = allOwners.iterator(); iter.hasNext(); ) {
- T owner = iter.next();
- // Get regions that are associated with this server and filter regions by group tables.
- for (RegionInfo region : getRegionsInfo.apply(owner)) {
- if (!validation.apply(region)) {
- LOG.info("Moving region {}, which do not belong to RSGroup {}",
- region.getShortNameToLog(), targetGroupName);
- try {
- this.master.getAssignmentManager().move(region);
- failedRegions.remove(region.getRegionNameAsString());
- } catch (IOException ioe) {
- LOG.debug("Move region {} from group failed, will retry, current retry time is {}",
- region.getShortNameToLog(), retry, ioe);
- toThrow = ioe;
- failedRegions.add(region.getRegionNameAsString());
- }
- if (master.getAssignmentManager().getRegionStates().
- getRegionState(region).isFailedOpen()) {
- continue;
- }
- hasRegionsToMove = true;
- }
- }
-
- if (!hasRegionsToMove) {
- LOG.info("No more regions to move from {} to RSGroup", getOwnerName.apply(owner));
- iter.remove();
- }
- }
-
- retry++;
- try {
- rsGroupInfoManager.wait(1000);
- } catch (InterruptedException e) {
- LOG.warn("Sleep interrupted", e);
- Thread.currentThread().interrupt();
- }
- } while (hasRegionsToMove && retry <= moveMaxRetry);
-
- //has up to max retry time or there are no more regions to move
- if (hasRegionsToMove) {
- // print failed moved regions, for later process conveniently
- String msg = String
- .format("move regions for group %s failed, failed regions: %s", targetGroupName,
- failedRegions);
- LOG.error(msg);
- throw new DoNotRetryIOException(
- msg + ", just record the last failed region's cause, more details in server log",
- toThrow);
- }
- }
-
- @Override
- public void moveServers(Set<Address> servers, String targetGroupName) throws IOException {
- if (servers == null) {
- throw new ConstraintException("The list of servers to move cannot be null.");
- }
- if (servers.isEmpty()) {
- // For some reason this difference between null servers and isEmpty is important distinction.
- // TODO. Why? Stuff breaks if I equate them.
- return;
- }
- //check target group
- getAndCheckRSGroupInfo(targetGroupName);
-
- // Hold a lock on the manager instance while moving servers to prevent
- // another writer changing our state while we are working.
- synchronized (rsGroupInfoManager) {
- // Presume first server's source group. Later ensure all servers are from this group.
- Address firstServer = servers.iterator().next();
- RSGroupInfo srcGrp = rsGroupInfoManager.getRSGroupOfServer(firstServer);
- if (srcGrp == null) {
- // Be careful. This exception message is tested for in TestRSGroupsBase...
- throw new ConstraintException("Source RSGroup for server " + firstServer
- + " does not exist.");
- }
- // Only move online servers (when moving from 'default') or servers from other
- // groups. This prevents bogus servers from entering groups
- if (RSGroupInfo.DEFAULT_GROUP.equals(srcGrp.getName())) {
- if (srcGrp.getServers().size() <= servers.size()) {
- throw new ConstraintException(KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE);
- }
- checkOnlineServersOnly(servers);
- }
- // Ensure all servers are of same rsgroup.
- for (Address server: servers) {
- String tmpGroup = rsGroupInfoManager.getRSGroupOfServer(server).getName();
- if (!tmpGroup.equals(srcGrp.getName())) {
- throw new ConstraintException("Move server request should only come from one source " +
- "RSGroup. Expecting only " + srcGrp.getName() + " but contains " + tmpGroup);
- }
- }
- if (srcGrp.getServers().size() <= servers.size()) {
- // check if there are still tables reference this group
- for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
- Optional<String> optGroupName = td.getRegionServerGroup();
- if (optGroupName.isPresent() && optGroupName.get().equals(srcGrp.getName())) {
- throw new ConstraintException(
- "Cannot leave a RSGroup " + srcGrp.getName() + " that contains tables('" +
- td.getTableName() + "' at least) without servers to host them.");
- }
- }
- }
-
- // MovedServers may be < passed in 'servers'.
- Set<Address> movedServers = rsGroupInfoManager.moveServers(servers, srcGrp.getName(),
- targetGroupName);
- moveServerRegionsFromGroup(movedServers, targetGroupName);
- LOG.info("Move servers done: {} => {}", srcGrp.getName(), targetGroupName);
- }
- }
-
- @Override
- public void addRSGroup(String name) throws IOException {
- rsGroupInfoManager.addRSGroup(new RSGroupInfo(name));
- }
-
- @Override
- public void removeRSGroup(String name) throws IOException {
- // Hold a lock on the manager instance while moving servers to prevent
- // another writer changing our state while we are working.
- synchronized (rsGroupInfoManager) {
- RSGroupInfo rsGroupInfo = rsGroupInfoManager.getRSGroup(name);
- if (rsGroupInfo == null) {
- throw new ConstraintException("RSGroup " + name + " does not exist");
- }
- int serverCount = rsGroupInfo.getServers().size();
- if (serverCount > 0) {
- throw new ConstraintException("RSGroup " + name + " has " + serverCount +
- " servers; you must remove these servers from the RSGroup before" +
- " the RSGroup can be removed.");
- }
- for (TableDescriptor td : master.getTableDescriptors().getAll().values()) {
- if (td.getRegionServerGroup().map(name::equals).orElse(false)) {
- throw new ConstraintException("RSGroup " + name + " is already referenced by " +
- td.getTableName() + "; you must remove all the tables from the rsgroup before " +
- "the rsgroup can be removed.");
- }
- }
- for (NamespaceDescriptor ns : master.getClusterSchema().getNamespaces()) {
- String nsGroup = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (nsGroup != null && nsGroup.equals(name)) {
- throw new ConstraintException(
- "RSGroup " + name + " is referenced by namespace: " + ns.getName());
- }
- }
- rsGroupInfoManager.removeRSGroup(name);
- }
- }
-
- @Override
- public boolean balanceRSGroup(String groupName) throws IOException {
- ServerManager serverManager = master.getServerManager();
- LoadBalancer balancer = master.getLoadBalancer();
-
- synchronized (balancer) {
- // If balance not true, don't run balancer.
- if (!((HMaster) master).isBalancerOn()) {
- return false;
- }
-
- if (getRSGroupInfo(groupName) == null) {
- throw new ConstraintException("RSGroup does not exist: " + groupName);
- }
- // Only allow one balance run at at time.
- Map<String, RegionState> groupRIT = rsGroupGetRegionsInTransition(groupName);
- if (groupRIT.size() > 0) {
- LOG.debug("Not running balancer because {} region(s) in transition: {}", groupRIT.size(),
- StringUtils.abbreviate(
- master.getAssignmentManager().getRegionStates().getRegionsInTransition().toString(),
- 256));
- return false;
- }
- if (serverManager.areDeadServersInProgress()) {
- LOG.debug("Not running balancer because processing dead regionserver(s): {}",
- serverManager.getDeadServers());
- return false;
- }
-
- // We balance per group instead of per table
- List<RegionPlan> plans = new ArrayList<>();
- Map<TableName, Map<ServerName, List<RegionInfo>>> assignmentsByTable =
- getRSGroupAssignmentsByTable(groupName);
- for (Map.Entry<TableName, Map<ServerName, List<RegionInfo>>> tableMap : assignmentsByTable
- .entrySet()) {
- LOG.info("Creating partial plan for table {} : {}", tableMap.getKey(), tableMap.getValue());
- List<RegionPlan> partialPlans = balancer.balanceCluster(tableMap.getValue());
- LOG.info("Partial plan for table {} : {}", tableMap.getKey(), partialPlans);
- if (partialPlans != null) {
- plans.addAll(partialPlans);
- }
- }
- boolean balancerRan = !plans.isEmpty();
- if (balancerRan) {
- LOG.info("RSGroup balance {} starting with plan count: {}", groupName, plans.size());
- master.executeRegionPlansWithThrottling(plans);
- LOG.info("RSGroup balance " + groupName + " completed");
- }
- return balancerRan;
- }
- }
-
- @Override
- public List<RSGroupInfo> listRSGroups() throws IOException {
- return rsGroupInfoManager.listRSGroups();
- }
-
- @Override
- public RSGroupInfo getRSGroupOfServer(Address hostPort) throws IOException {
- return rsGroupInfoManager.getRSGroupOfServer(hostPort);
- }
-
- @Override
- public void removeServers(Set<Address> servers) throws IOException {
- if (servers == null || servers.isEmpty()) {
- throw new ConstraintException("The set of servers to remove cannot be null or empty.");
- }
- // Hold a lock on the manager instance while moving servers to prevent
- // another writer changing our state while we are working.
- synchronized (rsGroupInfoManager) {
- // check the set of servers
- checkForDeadOrOnlineServers(servers);
- rsGroupInfoManager.removeServers(servers);
- LOG.info("Remove decommissioned servers {} from RSGroup done", servers);
- }
- }
-
- private boolean isTableInGroup(TableName tableName, String groupName,
- Set<TableName> tablesInGroupCache) throws IOException {
- if (tablesInGroupCache.contains(tableName)) {
- return true;
- }
- if (RSGroupUtil.getRSGroupInfo(master, rsGroupInfoManager, tableName).map(RSGroupInfo::getName)
- .orElse(RSGroupInfo.DEFAULT_GROUP).equals(groupName)) {
- tablesInGroupCache.add(tableName);
- return true;
- }
- return false;
- }
-
- private Map<String, RegionState> rsGroupGetRegionsInTransition(String groupName)
- throws IOException {
- Map<String, RegionState> rit = Maps.newTreeMap();
- Set<TableName> tablesInGroupCache = new HashSet<>();
- for (RegionStateNode regionNode : master.getAssignmentManager().getRegionsInTransition()) {
- TableName tn = regionNode.getTable();
- if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
- rit.put(regionNode.getRegionInfo().getEncodedName(), regionNode.toRegionState());
- }
- }
- return rit;
- }
-
- private Map<TableName, Map<ServerName, List<RegionInfo>>>
- getRSGroupAssignmentsByTable(String groupName) throws IOException {
- Map<TableName, Map<ServerName, List<RegionInfo>>> result = Maps.newHashMap();
- Set<TableName> tablesInGroupCache = new HashSet<>();
- for (Map.Entry<RegionInfo, ServerName> entry : master.getAssignmentManager().getRegionStates()
- .getRegionAssignments().entrySet()) {
- RegionInfo region = entry.getKey();
- TableName tn = region.getTable();
- ServerName server = entry.getValue();
- if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
- result.computeIfAbsent(tn, k -> new HashMap<>())
- .computeIfAbsent(server, k -> new ArrayList<>()).add(region);
- }
- }
- RSGroupInfo rsGroupInfo = getRSGroupInfo(groupName);
- for (ServerName serverName : master.getServerManager().getOnlineServers().keySet()) {
- if (rsGroupInfo.containsServer(serverName.getAddress())) {
- for (Map<ServerName, List<RegionInfo>> map : result.values()) {
- map.computeIfAbsent(serverName, k -> Collections.emptyList());
- }
- }
- }
-
- return result;
- }
-
- /**
- * Check if the set of servers are belong to dead servers list or online servers list.
- * @param servers servers to remove
- */
- private void checkForDeadOrOnlineServers(Set<Address> servers) throws ConstraintException {
- // This uglyness is because we only have Address, not ServerName.
- Set<Address> onlineServers = new HashSet<>();
- List<ServerName> drainingServers = master.getServerManager().getDrainingServersList();
- for (ServerName server : master.getServerManager().getOnlineServers().keySet()) {
- // Only online but not decommissioned servers are really online
- if (!drainingServers.contains(server)) {
- onlineServers.add(server.getAddress());
- }
- }
-
- Set<Address> deadServers = new HashSet<>();
- for(ServerName server: master.getServerManager().getDeadServers().copyServerNames()) {
- deadServers.add(server.getAddress());
- }
-
- for (Address address: servers) {
- if (onlineServers.contains(address)) {
- throw new ConstraintException(
- "Server " + address + " is an online server, not allowed to remove.");
- }
- if (deadServers.contains(address)) {
- throw new ConstraintException(
- "Server " + address + " is on the dead servers list,"
- + " Maybe it will come back again, not allowed to remove.");
- }
- }
- }
-}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
index 749d353..a00deec 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
@@ -33,7 +33,6 @@ import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
-import org.apache.hadoop.hbase.ipc.RpcServer;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait;
import org.apache.hadoop.hbase.net.Address;
@@ -63,62 +62,34 @@ import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGro
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupResponse;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersResponse;
-import org.apache.hadoop.hbase.security.User;
-import org.apache.hadoop.hbase.security.UserProvider;
-import org.apache.hadoop.hbase.security.access.AccessChecker;
-import org.apache.hadoop.hbase.security.access.Permission.Action;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
/**
* Implementation of RSGroupAdminService defined in RSGroupAdmin.proto. This class calls
- * {@link RSGroupAdminServer} for actual work, converts result to protocol buffer response, handles
- * exceptions if any occurred and then calls the {@code RpcCallback} with the response.
+ * {@link RSGroupInfoManagerImpl} for actual work, converts result to protocol buffer response,
+ * handles exceptions if any occurred and then calls the {@code RpcCallback} with the response.
+ *
+ * @deprecated Keep it here only for compatibility with {@link RSGroupAdminClient},
+ * using {@link org.apache.hadoop.hbase.master.MasterRpcServices} instead.
*/
+@Deprecated
class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
private static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminServiceImpl.class);
private MasterServices master;
- private RSGroupAdminServer groupAdminServer;
-
- private AccessChecker accessChecker;
-
- /** Provider for mapping principal names to Users */
- private UserProvider userProvider;
+ private RSGroupInfoManager rsGroupInfoManager;
RSGroupAdminServiceImpl() {
}
- void initialize(MasterServices master, RSGroupAdminServer groupAdminServer,
- AccessChecker accessChecker, UserProvider userProvider) {
- this.master = master;
- this.groupAdminServer = groupAdminServer;
- this.accessChecker = accessChecker;
- this.userProvider = userProvider;
- }
-
- @VisibleForTesting
- void checkPermission(String request) throws IOException {
- accessChecker.requirePermission(getActiveUser(), request, null, Action.ADMIN);
- }
-
- /**
- * Returns the active user to which authorization checks should be applied. If we are in the
- * context of an RPC call, the remote user is used, otherwise the currently logged in user is
- * used.
- */
- private User getActiveUser() throws IOException {
- // for non-rpc handling, fallback to system user
- Optional<User> optionalUser = RpcServer.getRequestUser();
- if (optionalUser.isPresent()) {
- return optionalUser.get();
- }
- return userProvider.getCurrent();
+ void initialize(MasterServices masterServices){
+ this.master = masterServices;
+ this.rsGroupInfoManager = masterServices.getRSGroupInfoManager();
}
// for backward compatible
@@ -137,8 +108,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preGetRSGroupInfo(groupName);
}
- checkPermission("getRSGroupInfo");
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
+ RSGroupInfo rsGroupInfo = rsGroupInfoManager.getRSGroup(groupName);
if (rsGroupInfo != null) {
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(rsGroupInfo)));
}
@@ -162,14 +132,13 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preGetRSGroupInfoOfTable(tableName);
}
- checkPermission("getRSGroupInfoOfTable");
Optional<RSGroupInfo> optGroup =
- RSGroupUtil.getRSGroupInfo(master, groupAdminServer.rsGroupInfoManager, tableName);
+ RSGroupUtil.getRSGroupInfo(master, rsGroupInfoManager, tableName);
if (optGroup.isPresent()) {
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(optGroup.get())));
} else {
if (master.getTableStateManager().isTablePresent(tableName)) {
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo rsGroupInfo = rsGroupInfoManager.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(rsGroupInfo)));
}
}
@@ -197,8 +166,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preMoveServers(hostPorts, request.getTargetGroup());
}
- checkPermission("moveServers");
- groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
+ rsGroupInfoManager.moveServers(hostPorts, request.getTargetGroup());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postMoveServers(hostPorts, request.getTargetGroup());
}
@@ -243,7 +211,6 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preMoveTables(tables, request.getTargetGroup());
}
- checkPermission("moveTables");
moveTablesAndWait(tables, request.getTargetGroup());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postMoveTables(tables, request.getTargetGroup());
@@ -263,8 +230,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
}
- checkPermission("addRSGroup");
- groupAdminServer.addRSGroup(request.getRSGroupName());
+ rsGroupInfoManager.addRSGroup(new RSGroupInfo(request.getRSGroupName()));
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postAddRSGroup(request.getRSGroupName());
}
@@ -283,8 +249,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preRemoveRSGroup(request.getRSGroupName());
}
- checkPermission("removeRSGroup");
- groupAdminServer.removeRSGroup(request.getRSGroupName());
+ rsGroupInfoManager.removeRSGroup(request.getRSGroupName());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postRemoveRSGroup(request.getRSGroupName());
}
@@ -304,8 +269,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preBalanceRSGroup(request.getRSGroupName());
}
- checkPermission("balanceRSGroup");
- boolean balancerRan = groupAdminServer.balanceRSGroup(request.getRSGroupName());
+ boolean balancerRan = rsGroupInfoManager.balanceRSGroup(request.getRSGroupName());
builder.setBalanceRan(balancerRan);
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postBalanceRSGroup(request.getRSGroupName(), balancerRan);
@@ -326,8 +290,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preListRSGroups();
}
- checkPermission("listRSGroup");
- List<RSGroupInfo> rsGroupInfos = groupAdminServer.listRSGroups().stream()
+ List<RSGroupInfo> rsGroupInfos = rsGroupInfoManager.listRSGroups().stream()
.map(RSGroupInfo::new).collect(Collectors.toList());
Map<String, RSGroupInfo> name2Info = new HashMap<>();
for (RSGroupInfo rsGroupInfo : rsGroupInfos) {
@@ -364,8 +327,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preGetRSGroupInfoOfServer(hp);
}
- checkPermission("getRSGroupInfoOfServer");
- RSGroupInfo info = groupAdminServer.getRSGroupOfServer(hp);
+ RSGroupInfo info = rsGroupInfoManager.getRSGroupOfServer(hp);
if (info != null) {
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(info)));
}
@@ -397,8 +359,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
master.getMasterCoprocessorHost().preMoveServersAndTables(hostPorts, tables,
request.getTargetGroup());
}
- checkPermission("moveServersAndTables");
- groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
+ rsGroupInfoManager.moveServers(hostPorts, request.getTargetGroup());
moveTablesAndWait(tables, request.getTargetGroup());
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postMoveServersAndTables(hostPorts, tables,
@@ -424,8 +385,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().preRemoveServers(servers);
}
- checkPermission("removeServers");
- groupAdminServer.removeServers(servers);
+ rsGroupInfoManager.removeServers(servers);
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postRemoveServers(servers);
}
@@ -434,4 +394,5 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
}
done.run(builder.build());
}
+
}
\ No newline at end of file
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 0eb15e9..c1b03a6 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -48,13 +48,8 @@ public interface RSGroupInfoManager {
/**
* Move servers to a new group.
- * @param servers list of servers, must be part of the same group
- * @param srcGroup groupName being moved from
- * @param dstGroup groupName being moved to
- * @return Set of servers moved (May be a subset of {@code servers}).
*/
- Set<Address> moveServers(Set<Address> servers, String srcGroup, String dstGroup)
- throws IOException;
+ void moveServers(Set<Address> servers, String targetGroupName) throws IOException;
/**
* Gets the group info of server.
@@ -85,10 +80,7 @@ public interface RSGroupInfoManager {
/**
* Get {@code RSGroupInfo} for the given table.
- * @deprecated Since 3.0.0, will be removed in 4.0.0. Only for compatibility, where we upgrade
- * from a version that stores table names for a rs group in the {@code RSGroupInfo}.
*/
- @Deprecated
RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException;
static RSGroupInfoManager create(MasterServices master) throws IOException {
@@ -98,4 +90,14 @@ public interface RSGroupInfoManager {
return new DisabledRSGroupInfoManager(master.getServerManager());
}
}
+
+ /**
+ * Balance a region server group.
+ */
+ boolean balanceRSGroup(String groupName) throws IOException;
+
+ /**
+ * Set group for tables.
+ */
+ void setRSGroup(Set<TableName> tables, String groupName) throws IOException;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index 7224869..83aa12e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -24,15 +24,21 @@ import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.OptionalLong;
import java.util.Set;
import java.util.SortedSet;
import java.util.TreeSet;
+import java.util.function.Function;
import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hbase.Coprocessor;
import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableDescriptors;
@@ -45,6 +51,7 @@ import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Mutation;
import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.TableDescriptor;
@@ -52,11 +59,17 @@ import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
+import org.apache.hadoop.hbase.master.LoadBalancer;
import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionPlan;
+import org.apache.hadoop.hbase.master.RegionState;
import org.apache.hadoop.hbase.master.ServerListener;
+import org.apache.hadoop.hbase.master.ServerManager;
import org.apache.hadoop.hbase.master.TableStateManager;
+import org.apache.hadoop.hbase.master.assignment.RegionStateNode;
import org.apache.hadoop.hbase.master.procedure.CreateTableProcedure;
import org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil;
+import org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait;
import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.procedure2.Procedure;
import org.apache.hadoop.hbase.protobuf.ProtobufMagic;
@@ -83,15 +96,13 @@ import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableMap;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
-
/**
* This is an implementation of {@link RSGroupInfoManager} which makes use of an HBase table as the
* persistence store for the group information. It also makes use of zookeeper to store group
* information needed for bootstrapping during offline mode.
* <h2>Concurrency</h2> RSGroup state is kept locally in Maps. There is a rsgroup name to cached
- * RSGroupInfo Map at {@link #rsGroupMap} and a Map of tables to the name of the rsgroup they belong
- * too (in {@link #tableMap}). These Maps are persisted to the hbase:rsgroup table (and cached in
- * zk) on each modification.
+ * RSGroupInfo Map at {@link #rsGroupMap}. These Maps are persisted to the hbase:rsgroup table
+ * (and cached in zk) on each modification.
* <p/>
* Mutations on state are synchronized but reads can continue without having to wait on an instance
* monitor, mutations do wholesale replace of the Maps on update -- Copy-On-Write; the local Maps of
@@ -113,6 +124,18 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
static final TableName RSGROUP_TABLE_NAME =
TableName.valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, "rsgroup");
+ @VisibleForTesting
+ static final String KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE = "should keep at least " +
+ "one server in 'default' RSGroup.";
+
+ /** Define the config key of retries threshold when movements failed */
+ @VisibleForTesting
+ static final String FAILED_MOVE_MAX_RETRY = "hbase.rsgroup.move.max.retry";
+
+ /** Define the default number of retries */
+ @VisibleForTesting
+ static final int DEFAULT_MAX_RETRY_VALUE = 50;
+
private static final String RS_GROUP_ZNODE = "rsgroup";
@VisibleForTesting
@@ -176,7 +199,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// contains list of groups that were last flushed to persistent store
private Set<String> prevRSGroups = new HashSet<>();
- private RSGroupInfoManagerImpl(MasterServices masterServices) throws IOException {
+ private RSGroupInfoManagerImpl(MasterServices masterServices) {
this.masterServices = masterServices;
this.watcher = masterServices.getZooKeeper();
this.conn = masterServices.getAsyncClusterConnection();
@@ -214,8 +237,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
migrate();
}
- static RSGroupInfoManager getInstance(MasterServices master) throws IOException {
- RSGroupInfoManagerImpl instance = new RSGroupInfoManagerImpl(master);
+ static RSGroupInfoManager getInstance(MasterServices masterServices) throws IOException {
+ RSGroupInfoManagerImpl instance = new RSGroupInfoManagerImpl(masterServices);
instance.init();
return instance;
}
@@ -231,17 +254,17 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
if (rsGroupMap.get(rsGroupInfo.getName()) != null ||
rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
- throw new DoNotRetryIOException("Group already exists: " + rsGroupInfo.getName());
+ throw new ConstraintException("Group already exists: " + rsGroupInfo.getName());
}
Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
newGroupMap.put(rsGroupInfo.getName(), rsGroupInfo);
flushConfig(newGroupMap);
}
- private RSGroupInfo getRSGroupInfo(final String groupName) throws DoNotRetryIOException {
- RSGroupInfo rsGroupInfo = getRSGroup(groupName);
+ private RSGroupInfo getRSGroupInfo(final String groupName) throws ConstraintException {
+ RSGroupInfo rsGroupInfo = holder.groupName2Group.get(groupName);
if (rsGroupInfo == null) {
- throw new DoNotRetryIOException("RSGroup " + groupName + " does not exist");
+ throw new ConstraintException("RSGroup " + groupName + " does not exist");
}
return rsGroupInfo;
}
@@ -254,7 +277,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
.map(ServerName::getAddress).collect(Collectors.toSet());
}
- @Override
public synchronized Set<Address> moveServers(Set<Address> servers, String srcGroup,
String dstGroup) throws IOException {
RSGroupInfo src = getRSGroupInfo(srcGroup);
@@ -284,7 +306,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
@Override
- public RSGroupInfo getRSGroupOfServer(Address serverHostPort) throws IOException {
+ public RSGroupInfo getRSGroupOfServer(Address serverHostPort) {
for (RSGroupInfo info : holder.groupName2Group.values()) {
if (info.containsServer(serverHostPort)) {
return info;
@@ -300,9 +322,30 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public synchronized void removeRSGroup(String groupName) throws IOException {
+ RSGroupInfo rsGroupInfo = getRSGroupInfo(groupName);
+ int serverCount = rsGroupInfo.getServers().size();
+ if (serverCount > 0) {
+ throw new ConstraintException("RSGroup " + groupName + " has " + serverCount +
+ " servers; you must remove these servers from the RSGroup before" +
+ " the RSGroup can be removed.");
+ }
+ for (TableDescriptor td : masterServices.getTableDescriptors().getAll().values()) {
+ if (td.getRegionServerGroup().map(groupName::equals).orElse(false)) {
+ throw new ConstraintException("RSGroup " + groupName + " is already referenced by " +
+ td.getTableName() + "; you must remove all the tables from the rsgroup before " +
+ "the rsgroup can be removed.");
+ }
+ }
+ for (NamespaceDescriptor ns : masterServices.getClusterSchema().getNamespaces()) {
+ String nsGroup = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (nsGroup != null && nsGroup.equals(groupName)) {
+ throw new ConstraintException(
+ "RSGroup " + groupName + " is referenced by namespace: " + ns.getName());
+ }
+ }
Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
if (!rsGroupMap.containsKey(groupName) || groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
- throw new DoNotRetryIOException(
+ throw new ConstraintException(
"Group " + groupName + " does not exist or is a reserved " + "group");
}
Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
@@ -322,6 +365,13 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public synchronized void removeServers(Set<Address> servers) throws IOException {
+ if (servers == null || servers.isEmpty()) {
+ throw new ConstraintException("The set of servers to remove cannot be null or empty.");
+ }
+
+ // check the set of servers
+ checkForDeadOrOnlineServers(servers);
+
Map<String, RSGroupInfo> rsGroupInfos = new HashMap<String, RSGroupInfo>();
for (Address el : servers) {
RSGroupInfo rsGroupInfo = getRSGroupOfServer(el);
@@ -344,6 +394,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
newGroupMap.putAll(rsGroupInfos);
flushConfig(newGroupMap);
}
+ LOG.info("Remove decommissioned servers {} from RSGroup done", servers);
}
private List<RSGroupInfo> retrieveGroupListFromGroupTable() throws IOException {
@@ -575,7 +626,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Make changes visible after having been persisted to the source of truth
resetRSGroupMap(newGroupMap);
saveRSGroupMapToZK(newGroupMap);
-
updateCacheOfRSGroups(newGroupMap.keySet());
}
@@ -764,9 +814,371 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
}
-
@Override
public RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException {
return holder.tableName2Group.get(tableName);
}
+
+
+ /**
+ * Check if the set of servers are belong to dead servers list or online servers list.
+ * @param servers servers to remove
+ */
+ private void checkForDeadOrOnlineServers(Set<Address> servers) throws IOException {
+ // This uglyness is because we only have Address, not ServerName.
+ Set<Address> onlineServers = new HashSet<>();
+ List<ServerName> drainingServers = masterServices.getServerManager().getDrainingServersList();
+ for (ServerName server : masterServices.getServerManager().getOnlineServers().keySet()) {
+ // Only online but not decommissioned servers are really online
+ if (!drainingServers.contains(server)) {
+ onlineServers.add(server.getAddress());
+ }
+ }
+
+ Set<Address> deadServers = new HashSet<>();
+ for(ServerName server: masterServices.getServerManager().getDeadServers().copyServerNames()) {
+ deadServers.add(server.getAddress());
+ }
+
+ for (Address address: servers) {
+ if (onlineServers.contains(address)) {
+ throw new DoNotRetryIOException(
+ "Server " + address + " is an online server, not allowed to remove.");
+ }
+ if (deadServers.contains(address)) {
+ throw new DoNotRetryIOException(
+ "Server " + address + " is on the dead servers list,"
+ + " Maybe it will come back again, not allowed to remove.");
+ }
+ }
+ }
+
+ private void checkOnlineServersOnly(Set<Address> servers) throws IOException {
+ // This uglyness is because we only have Address, not ServerName.
+ // Online servers are keyed by ServerName.
+ Set<Address> onlineServers = new HashSet<>();
+ for(ServerName server: masterServices.getServerManager().getOnlineServers().keySet()) {
+ onlineServers.add(server.getAddress());
+ }
+ for (Address address: servers) {
+ if (!onlineServers.contains(address)) {
+ throw new DoNotRetryIOException("Server " + address +
+ " is not an online server in 'default' RSGroup.");
+ }
+ }
+ }
+
+ /**
+ * @return List of Regions associated with this <code>server</code>.
+ */
+ private List<RegionInfo> getRegions(final Address server) {
+ LinkedList<RegionInfo> regions = new LinkedList<>();
+ for (Map.Entry<RegionInfo, ServerName> el :
+ masterServices.getAssignmentManager().getRegionStates().getRegionAssignments().entrySet()) {
+ if (el.getValue() == null) {
+ continue;
+ }
+
+ if (el.getValue().getAddress().equals(server)) {
+ addRegion(regions, el.getKey());
+ }
+ }
+ for (RegionStateNode state : masterServices.getAssignmentManager().getRegionsInTransition()) {
+ if (state.getRegionLocation() != null &&
+ state.getRegionLocation().getAddress().equals(server)) {
+ addRegion(regions, state.getRegionInfo());
+ }
+ }
+ return regions;
+ }
+
+ private void addRegion(final LinkedList<RegionInfo> regions, RegionInfo hri) {
+ // If meta, move it last otherwise other unassigns fail because meta is not
+ // online for them to update state in. This is dodgy. Needs to be made more
+ // robust. See TODO below.
+ if (hri.isMetaRegion()) {
+ regions.addLast(hri);
+ } else {
+ regions.addFirst(hri);
+ }
+ }
+
+ /**
+ * Move every region from servers which are currently located on these servers, but should not be
+ * located there.
+ * @param servers the servers that will move to new group
+ * @param targetGroupName the target group name
+ * @throws IOException if moving the server and tables fail
+ */
+ private void moveServerRegionsFromGroup(Set<Address> servers, String targetGroupName)
+ throws IOException {
+ moveRegionsBetweenGroups(servers, targetGroupName, rs -> getRegions(rs), info -> {
+ try {
+ String groupName = RSGroupUtil.getRSGroupInfo(masterServices, this, info.getTable())
+ .map(RSGroupInfo::getName).orElse(RSGroupInfo.DEFAULT_GROUP);
+ return groupName.equals(targetGroupName);
+ } catch (IOException e) {
+ LOG.warn("Failed to test group for region {} and target group {}", info, targetGroupName);
+ return false;
+ }
+ }, rs -> rs.getHostname());
+ }
+
+ private <T> void moveRegionsBetweenGroups(Set<T> regionsOwners, String targetGroupName,
+ Function<T, List<RegionInfo>> getRegionsInfo, Function<RegionInfo, Boolean> validation,
+ Function<T, String> getOwnerName) throws IOException {
+ boolean hasRegionsToMove;
+ int retry = 0;
+ Set<T> allOwners = new HashSet<>(regionsOwners);
+ Set<String> failedRegions = new HashSet<>();
+ IOException toThrow = null;
+ do {
+ hasRegionsToMove = false;
+ for (Iterator<T> iter = allOwners.iterator(); iter.hasNext(); ) {
+ T owner = iter.next();
+ // Get regions that are associated with this server and filter regions by group tables.
+ for (RegionInfo region : getRegionsInfo.apply(owner)) {
+ if (!validation.apply(region)) {
+ LOG.info("Moving region {}, which do not belong to RSGroup {}",
+ region.getShortNameToLog(), targetGroupName);
+ try {
+ this.masterServices.getAssignmentManager().move(region);
+ failedRegions.remove(region.getRegionNameAsString());
+ } catch (IOException ioe) {
+ LOG.debug("Move region {} from group failed, will retry, current retry time is {}",
+ region.getShortNameToLog(), retry, ioe);
+ toThrow = ioe;
+ failedRegions.add(region.getRegionNameAsString());
+ }
+ if (masterServices.getAssignmentManager().getRegionStates().
+ getRegionState(region).isFailedOpen()) {
+ continue;
+ }
+ hasRegionsToMove = true;
+ }
+ }
+
+ if (!hasRegionsToMove) {
+ LOG.info("No more regions to move from {} to RSGroup", getOwnerName.apply(owner));
+ iter.remove();
+ }
+ }
+
+ retry++;
+ try {
+ wait(1000);
+ } catch (InterruptedException e) {
+ LOG.warn("Sleep interrupted", e);
+ Thread.currentThread().interrupt();
+ }
+ } while (hasRegionsToMove && retry <=
+ masterServices.getConfiguration().getInt(FAILED_MOVE_MAX_RETRY, DEFAULT_MAX_RETRY_VALUE));
+
+ //has up to max retry time or there are no more regions to move
+ if (hasRegionsToMove) {
+ // print failed moved regions, for later process conveniently
+ String msg = String
+ .format("move regions for group %s failed, failed regions: %s", targetGroupName,
+ failedRegions);
+ LOG.error(msg);
+ throw new DoNotRetryIOException(
+ msg + ", just record the last failed region's cause, more details in server log",
+ toThrow);
+ }
+ }
+
+ private boolean isTableInGroup(TableName tableName, String groupName,
+ Set<TableName> tablesInGroupCache) throws IOException {
+ if (tablesInGroupCache.contains(tableName)) {
+ return true;
+ }
+ if (RSGroupUtil.getRSGroupInfo(masterServices, this, tableName)
+ .map(RSGroupInfo::getName)
+ .orElse(RSGroupInfo.DEFAULT_GROUP).equals(groupName)) {
+ tablesInGroupCache.add(tableName);
+ return true;
+ }
+ return false;
+ }
+
+ private Map<String, RegionState> rsGroupGetRegionsInTransition(String groupName)
+ throws IOException {
+ Map<String, RegionState> rit = Maps.newTreeMap();
+ Set<TableName> tablesInGroupCache = new HashSet<>();
+ for (RegionStateNode regionNode :
+ masterServices.getAssignmentManager().getRegionsInTransition()) {
+ TableName tn = regionNode.getTable();
+ if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
+ rit.put(regionNode.getRegionInfo().getEncodedName(), regionNode.toRegionState());
+ }
+ }
+ return rit;
+ }
+
+ private Map<TableName, Map<ServerName, List<RegionInfo>>> getRSGroupAssignmentsByTable(
+ String groupName) throws IOException {
+ Map<TableName, Map<ServerName, List<RegionInfo>>> result = Maps.newHashMap();
+ Set<TableName> tablesInGroupCache = new HashSet<>();
+ for (Map.Entry<RegionInfo, ServerName> entry :
+ masterServices.getAssignmentManager().getRegionStates()
+ .getRegionAssignments().entrySet()) {
+ RegionInfo region = entry.getKey();
+ TableName tn = region.getTable();
+ ServerName server = entry.getValue();
+ if (isTableInGroup(tn, groupName, tablesInGroupCache)) {
+ result.computeIfAbsent(tn, k -> new HashMap<>())
+ .computeIfAbsent(server, k -> new ArrayList<>()).add(region);
+ }
+ }
+ RSGroupInfo rsGroupInfo = getRSGroupInfo(groupName);
+ for (ServerName serverName : masterServices.getServerManager().getOnlineServers().keySet()) {
+ if (rsGroupInfo.containsServer(serverName.getAddress())) {
+ for (Map<ServerName, List<RegionInfo>> map : result.values()) {
+ map.computeIfAbsent(serverName, k -> Collections.emptyList());
+ }
+ }
+ }
+
+ return result;
+ }
+
+ @Override
+ public boolean balanceRSGroup(String groupName) throws IOException {
+ ServerManager serverManager = masterServices.getServerManager();
+ LoadBalancer balancer = masterServices.getLoadBalancer();
+ getRSGroupInfo(groupName);
+
+ synchronized (balancer) {
+ // If balance not true, don't run balancer.
+ if (!masterServices.isBalancerOn()) {
+ return false;
+ }
+ // Only allow one balance run at at time.
+ Map<String, RegionState> groupRIT = rsGroupGetRegionsInTransition(groupName);
+ if (groupRIT.size() > 0) {
+ LOG.debug("Not running balancer because {} region(s) in transition: {}", groupRIT.size(),
+ StringUtils.abbreviate(masterServices.getAssignmentManager().getRegionStates()
+ .getRegionsInTransition().toString(),
+ 256));
+ return false;
+ }
+ if (serverManager.areDeadServersInProgress()) {
+ LOG.debug("Not running balancer because processing dead regionserver(s): {}",
+ serverManager.getDeadServers());
+ return false;
+ }
+
+ // We balance per group instead of per table
+ List<RegionPlan> plans = new ArrayList<>();
+ Map<TableName, Map<ServerName, List<RegionInfo>>> assignmentsByTable =
+ getRSGroupAssignmentsByTable(groupName);
+ for (Map.Entry<TableName, Map<ServerName, List<RegionInfo>>> tableMap : assignmentsByTable
+ .entrySet()) {
+ LOG.info("Creating partial plan for table {} : {}", tableMap.getKey(), tableMap.getValue());
+ List<RegionPlan> partialPlans = balancer.balanceCluster(tableMap.getValue());
+ LOG.info("Partial plan for table {} : {}", tableMap.getKey(), partialPlans);
+ if (partialPlans != null) {
+ plans.addAll(partialPlans);
+ }
+ }
+ boolean balancerRan = !plans.isEmpty();
+ if (balancerRan) {
+ LOG.info("RSGroup balance {} starting with plan count: {}", groupName, plans.size());
+ masterServices.executeRegionPlansWithThrottling(plans);
+ LOG.info("RSGroup balance " + groupName + " completed");
+ }
+ return balancerRan;
+ }
+ }
+
+ private void moveTablesAndWait(Set<TableName> tables, String targetGroup) throws IOException {
+ List<Long> procIds = new ArrayList<Long>();
+ for (TableName tableName : tables) {
+ TableDescriptor oldTd = masterServices.getTableDescriptors().get(tableName);
+ if (oldTd == null) {
+ continue;
+ }
+ TableDescriptor newTd =
+ TableDescriptorBuilder.newBuilder(oldTd).setRegionServerGroup(targetGroup).build();
+ procIds.add(masterServices.modifyTable(tableName, newTd, HConstants.NO_NONCE,
+ HConstants.NO_NONCE));
+ }
+ for (long procId : procIds) {
+ Procedure<?> proc = masterServices.getMasterProcedureExecutor().getProcedure(procId);
+ if (proc == null) {
+ continue;
+ }
+ ProcedureSyncWait.waitForProcedureToCompleteIOE(masterServices.getMasterProcedureExecutor(),
+ proc, Long.MAX_VALUE);
+ }
+ }
+
+ @Override
+ public void setRSGroup(Set<TableName> tables, String groupName) throws IOException {
+ getRSGroupInfo(groupName);
+ moveTablesAndWait(tables, groupName);
+ }
+
+ public void moveServers(Set<Address> servers, String targetGroupName) throws IOException {
+ if (servers == null) {
+ throw new ConstraintException("The list of servers to move cannot be null.");
+ }
+ if (servers.isEmpty()) {
+ // For some reason this difference between null servers and isEmpty is important distinction.
+ // TODO. Why? Stuff breaks if I equate them.
+ return;
+ }
+ if (StringUtils.isEmpty(targetGroupName)) {
+ throw new ConstraintException("RSGroup cannot be null.");
+ }
+ getRSGroupInfo(targetGroupName);
+
+ // Hold a lock on the manager instance while moving servers to prevent
+ // another writer changing our state while we are working.
+ synchronized (this) {
+ // Presume first server's source group. Later ensure all servers are from this group.
+ Address firstServer = servers.iterator().next();
+ RSGroupInfo srcGrp = getRSGroupOfServer(firstServer);
+ if (srcGrp == null) {
+ // Be careful. This exception message is tested for in TestRSGroupsBase...
+ throw new ConstraintException("Source RSGroup for server " + firstServer
+ + " does not exist.");
+ }
+
+ // Only move online servers (when moving from 'default') or servers from other
+ // groups. This prevents bogus servers from entering groups
+ if (RSGroupInfo.DEFAULT_GROUP.equals(srcGrp.getName())) {
+ if (srcGrp.getServers().size() <= servers.size()) {
+ throw new ConstraintException(KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE);
+ }
+ checkOnlineServersOnly(servers);
+ }
+ // Ensure all servers are of same rsgroup.
+ for (Address server: servers) {
+ String tmpGroup = getRSGroupOfServer(server).getName();
+ if (!tmpGroup.equals(srcGrp.getName())) {
+ throw new ConstraintException("Move server request should only come from one source " +
+ "RSGroup. Expecting only " + srcGrp.getName() + " but contains " + tmpGroup);
+ }
+ }
+ if (srcGrp.getServers().size() <= servers.size()) {
+ // check if there are still tables reference this group
+ for (TableDescriptor td : masterServices.getTableDescriptors().getAll().values()) {
+ Optional<String> optGroupName = td.getRegionServerGroup();
+ if (optGroupName.isPresent() && optGroupName.get().equals(srcGrp.getName())) {
+ throw new ConstraintException(
+ "Cannot leave a RSGroup " + srcGrp.getName() + " that contains tables('" +
+ td.getTableName() + "' at least) without servers to host them.");
+ }
+ }
+ }
+
+ // MovedServers may be < passed in 'servers'.
+ Set<Address> movedServers = moveServers(servers, srcGrp.getName(),
+ targetGroupName);
+ moveServerRegionsFromGroup(movedServers, targetGroupName);
+ LOG.info("Move servers done: {} => {}", srcGrp.getName(), targetGroupName);
+ }
+ }
+
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
index d1b3751..fa07108 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
@@ -19,11 +19,11 @@
package org.apache.hadoop.hbase.rsgroup;
import java.util.Arrays;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HBaseInterfaceAudience;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.util.compaction.MajorCompactorTTL;
@@ -31,6 +31,7 @@ import org.apache.hadoop.util.ToolRunner;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLine;
import org.apache.hbase.thirdparty.org.apache.commons.cli.CommandLineParser;
@@ -58,9 +59,9 @@ public class RSGroupMajorCompactionTTL extends MajorCompactorTTL {
throws Exception {
Connection conn = ConnectionFactory.createConnection(conf);
- RSGroupAdmin rsGroupAdmin = new RSGroupAdminClient(conn);
+ Admin admin = conn.getAdmin();
- RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroupInfo(rsgroup);
+ RSGroupInfo rsGroupInfo = admin.getRSGroup(rsgroup);
if (rsGroupInfo == null) {
LOG.error("Invalid rsgroup specified: " + rsgroup);
throw new IllegalArgumentException("Invalid rsgroup specified: " + rsgroup);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
index af30049..8aef0a1 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
@@ -27,7 +27,7 @@ import org.slf4j.LoggerFactory;
* Helper class for RSGroup implementation
*/
@InterfaceAudience.Private
-final class RSGroupUtil {
+public final class RSGroupUtil {
private static final Logger LOG = LoggerFactory.getLogger(RSGroupUtil.class);
@@ -38,8 +38,8 @@ final class RSGroupUtil {
* Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
* from the {@link NamespaceDescriptor}. If still not present, return empty.
*/
- static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupInfoManager manager,
- TableName tableName) throws IOException {
+ public static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master,
+ RSGroupInfoManager manager, TableName tableName) throws IOException {
TableDescriptor td = master.getTableDescriptors().get(tableName);
if (td == null) {
return Optional.empty();
@@ -79,7 +79,7 @@ final class RSGroupUtil {
* Fill the tables field for {@link RSGroupInfo}, for backward compatibility.
*/
@SuppressWarnings("deprecation")
- static RSGroupInfo fillTables(RSGroupInfo rsGroupInfo, Collection<TableDescriptor> tds) {
+ public static RSGroupInfo fillTables(RSGroupInfo rsGroupInfo, Collection<TableDescriptor> tds) {
RSGroupInfo newRsGroupInfo = new RSGroupInfo(rsGroupInfo);
Predicate<TableDescriptor> filter;
if (rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
index 44363fa..d46b88d 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
@@ -98,6 +98,7 @@ import org.apache.hadoop.hbase.io.hfile.HFile;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
import org.apache.hadoop.hbase.ipc.RpcServer;
import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos;
import org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.AccessControlService;
@@ -2588,4 +2589,84 @@ public class AccessController implements MasterCoprocessor, RegionCoprocessor,
}
}
}
+
+ @Override
+ public void preMoveServersAndTables(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ Set<Address> servers, Set<TableName> tables, String targetGroup) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "moveServersAndTables",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preMoveServers(final ObserverContext<MasterCoprocessorEnvironment> ctx,
+ Set<Address> servers, String targetGroup) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "moveServers",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preMoveTables(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ Set<TableName> tables, String targetGroup) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "moveTables",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preAddRSGroup(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ String name) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "addRSGroup",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preRemoveRSGroup(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ String name) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "removeRSGroup",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preBalanceRSGroup(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ String groupName) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "balanceRSGroup",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preRemoveServers(
+ ObserverContext<MasterCoprocessorEnvironment> ctx,
+ Set<Address> servers) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "removeServers",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preGetRSGroupInfo(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ String groupName) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "getRSGroupInfo",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preGetRSGroupInfoOfTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ TableName tableName) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "getRSGroupInfoOfTable",
+ null, Permission.Action.ADMIN);
+ //todo: should add check for table existence
+ }
+
+ @Override
+ public void preListRSGroups(ObserverContext<MasterCoprocessorEnvironment> ctx)
+ throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "listRSGroups",
+ null, Permission.Action.ADMIN);
+ }
+
+ @Override
+ public void preGetRSGroupInfoOfServer(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ Address server) throws IOException {
+ accessChecker.requirePermission(getActiveUser(ctx), "getRSGroupInfoOfServer",
+ null, Permission.Action.ADMIN);
+ }
+
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
index 71f8a91..c2640c9 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
@@ -496,4 +496,9 @@ public class MockNoopMasterServices implements MasterServices {
public RSGroupInfoManager getRSGroupInfoManager() {
return null;
}
+
+ @Override
+ public boolean isBalancerOn() {
+ return false;
+ }
}
\ No newline at end of file
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java
index f61e18a..122abc4 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java
@@ -41,15 +41,18 @@ import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.zookeeper.KeeperException;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
+import org.junit.After;
+import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
/**
* Testcase for HBASE-22819
*/
+@RunWith(Parameterized.class)
@Category({ MediumTests.class })
public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
@@ -63,8 +66,8 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
private static byte[] FAMILY = Bytes.toBytes("family");
- @BeforeClass
- public static void setUp() throws Exception {
+ @Before
+ public void setUp() throws Exception {
TEST_UTIL.getConfiguration().setClass(HConstants.MASTER_IMPL, HMasterForTest.class,
HMaster.class);
setUpTestBeforeClass();
@@ -73,8 +76,8 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
}
}
- @AfterClass
- public static void tearDown() throws Exception {
+ @After
+ public void tearDown() throws Exception {
tearDownAfterClass();
}
@@ -90,7 +93,7 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
public TableDescriptors getTableDescriptors() {
if (RESUME != null) {
for (StackTraceElement element : Thread.currentThread().getStackTrace()) {
- if (element.getClassName().contains("RSGroupInfoManagerImpl")) {
+ if (element.getMethodName().equals("migrate")) {
try {
RESUME.await();
} catch (InterruptedException e) {
@@ -106,9 +109,10 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
@Test
public void testMigrate() throws IOException, InterruptedException {
- String groupName = name.getMethodName();
+ setAdmin();
+ String groupName = getNameWithoutIndex(name.getMethodName());
addGroup(groupName, TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().size() - 1);
- RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroupInfo(groupName);
+ RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroup(groupName);
assertTrue(rsGroupInfo.getTables().isEmpty());
for (int i = 0; i < NUM_TABLES; i++) {
rsGroupInfo.addTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
@@ -126,7 +130,7 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
// wait until we can get the rs group info for a table
TEST_UTIL.waitFor(30000, () -> {
try {
- rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + 0));
+ rsGroupAdmin.getRSGroup(TableName.valueOf(TABLE_NAME_PREFIX + 0));
return true;
} catch (IOException e) {
return false;
@@ -135,7 +139,7 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
// confirm that before migrating, we could still get the correct rs group for a table.
for (int i = 0; i < NUM_TABLES; i++) {
RSGroupInfo info =
- rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ rsGroupAdmin.getRSGroup(TableName.valueOf(TABLE_NAME_PREFIX + i));
assertEquals(rsGroupInfo.getName(), info.getName());
assertEquals(NUM_TABLES, info.getTables().size());
}
@@ -171,7 +175,7 @@ public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
// make sure we could still get the correct rs group info after migration
for (int i = 0; i < NUM_TABLES; i++) {
RSGroupInfo info =
- rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ rsGroupAdmin.getRSGroup(TableName.valueOf(TABLE_NAME_PREFIX + i));
assertEquals(rsGroupInfo.getName(), info.getName());
assertEquals(NUM_TABLES, info.getTables().size());
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
index aff591f..614b6c4 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
@@ -42,11 +42,8 @@ import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.constraint.ConstraintException;
-import org.apache.hadoop.hbase.master.ServerManager;
import org.apache.hadoop.hbase.master.TableNamespaceManager;
-import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
import org.apache.hadoop.hbase.net.Address;
-import org.apache.hadoop.hbase.quotas.QuotaUtil;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.After;
@@ -57,11 +54,14 @@ import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
+@RunWith(Parameterized.class)
@Category({ MediumTests.class })
public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@@ -112,9 +112,9 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@Test
public void testBogusArgs() throws Exception {
- assertNull(rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf("nonexistent")));
- assertNull(rsGroupAdmin.getRSGroupOfServer(Address.fromParts("bogus", 123)));
- assertNull(rsGroupAdmin.getRSGroupInfo("bogus"));
+ assertNull(rsGroupAdmin.getRSGroup(TableName.valueOf("nonexistent")));
+ assertNull(rsGroupAdmin.getRSGroup(Address.fromParts("bogus", 123)));
+ assertNull(rsGroupAdmin.getRSGroup("bogus"));
try {
rsGroupAdmin.removeRSGroup("bogus");
@@ -124,14 +124,15 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
try {
- rsGroupAdmin.moveTables(Sets.newHashSet(TableName.valueOf("bogustable")), "bogus");
- fail("Expected move with bogus group to fail");
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(TableName.valueOf("bogustable")), "bogus");
+ fail("Expected set table to bogus group fail");
} catch (ConstraintException | TableNotFoundException ex) {
// expected
}
try {
- rsGroupAdmin.moveServers(Sets.newHashSet(Address.fromParts("bogus", 123)), "bogus");
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(Address.fromParts("bogus", 123)),
+ "bogus");
fail("Expected move with bogus group to fail");
} catch (ConstraintException ex) {
// expected
@@ -158,8 +159,8 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
admin.createNamespace(NamespaceDescriptor.create(nsName)
.addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, groupName).build());
- RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroupInfo(groupName);
- rsGroupAdmin.moveServers(rsGroupInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroup(groupName);
+ rsGroupAdmin.moveToRSGroup(rsGroupInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
// test removing a referenced group
try {
rsGroupAdmin.removeRSGroup(groupName);
@@ -198,16 +199,20 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@Test
public void testGetRSGroupInfoCPHookCalled() throws Exception {
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
- assertTrue(observer.preGetRSGroupInfoCalled);
- assertTrue(observer.postGetRSGroupInfoCalled);
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
+ if (rsGroupAdmin instanceof RSGroupAdminClient) {
+ assertTrue(observer.preGetRSGroupInfoCalled);
+ assertTrue(observer.postGetRSGroupInfoCalled);
+ }
}
@Test
public void testGetRSGroupInfoOfTableCPHookCalled() throws Exception {
- rsGroupAdmin.getRSGroupInfoOfTable(TableName.META_TABLE_NAME);
- assertTrue(observer.preGetRSGroupInfoOfTableCalled);
- assertTrue(observer.postGetRSGroupInfoOfTableCalled);
+ rsGroupAdmin.getRSGroup(TableName.META_TABLE_NAME);
+ if (rsGroupAdmin instanceof RSGroupAdminClient) {
+ assertTrue(observer.preGetRSGroupInfoOfTableCalled);
+ assertTrue(observer.postGetRSGroupInfoOfTableCalled);
+ }
}
@Test
@@ -220,9 +225,11 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@Test
public void testGetRSGroupInfoOfServerCPHookCalled() throws Exception {
ServerName masterServerName = ((MiniHBaseCluster) cluster).getMaster().getServerName();
- rsGroupAdmin.getRSGroupOfServer(masterServerName.getAddress());
- assertTrue(observer.preGetRSGroupInfoOfServerCalled);
- assertTrue(observer.postGetRSGroupInfoOfServerCalled);
+ rsGroupAdmin.getRSGroup(masterServerName.getAddress());
+ if (rsGroupAdmin instanceof RSGroupAdminClient) {
+ assertTrue(observer.preGetRSGroupInfoOfServerCalled);
+ assertTrue(observer.postGetRSGroupInfoOfServerCalled);
+ }
}
@Test
@@ -230,8 +237,8 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
int initNumGroups = rsGroupAdmin.listRSGroups().size();
addGroup("bar", 3);
TEST_UTIL.createTable(tableName, Bytes.toBytes("f"));
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), "bar");
- RSGroupInfo barGroup = rsGroupAdmin.getRSGroupInfo("bar");
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), "bar");
+ RSGroupInfo barGroup = rsGroupAdmin.getRSGroup("bar");
// group is not empty therefore it should fail
try {
rsGroupAdmin.removeRSGroup(barGroup.getName());
@@ -240,19 +247,19 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
// group cannot lose all it's servers therefore it should fail
try {
- rsGroupAdmin.moveServers(barGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(barGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
fail("Expected move servers to fail");
} catch (IOException e) {
}
- rsGroupAdmin.moveTables(barGroup.getTables(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.setRSGroup(barGroup.getTables(), RSGroupInfo.DEFAULT_GROUP);
try {
rsGroupAdmin.removeRSGroup(barGroup.getName());
fail("Expected move servers to fail");
} catch (IOException e) {
}
- rsGroupAdmin.moveServers(barGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(barGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.removeRSGroup(barGroup.getName());
Assert.assertEquals(initNumGroups, rsGroupAdmin.listRSGroups().size());
@@ -260,10 +267,12 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@Test
public void testMultiTableMove() throws Exception {
- final TableName tableNameA = TableName.valueOf(tablePrefix + name.getMethodName() + "A");
- final TableName tableNameB = TableName.valueOf(tablePrefix + name.getMethodName() + "B");
+ final TableName tableNameA = TableName.valueOf(tablePrefix +
+ getNameWithoutIndex(name.getMethodName()) + "A");
+ final TableName tableNameB = TableName.valueOf(tablePrefix +
+ getNameWithoutIndex(name.getMethodName()) + "B");
final byte[] familyNameBytes = Bytes.toBytes("f");
- String newGroupName = getGroupName(name.getMethodName());
+ String newGroupName = getGroupName(getNameWithoutIndex(name.getMethodName()));
final RSGroupInfo newGroup = addGroup(newGroupName, 1);
TEST_UTIL.createTable(tableNameA, familyNameBytes);
@@ -284,30 +293,30 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
});
- RSGroupInfo tableGrpA = rsGroupAdmin.getRSGroupInfoOfTable(tableNameA);
+ RSGroupInfo tableGrpA = rsGroupAdmin.getRSGroup(tableNameA);
assertTrue(tableGrpA.getName().equals(RSGroupInfo.DEFAULT_GROUP));
- RSGroupInfo tableGrpB = rsGroupAdmin.getRSGroupInfoOfTable(tableNameB);
+ RSGroupInfo tableGrpB = rsGroupAdmin.getRSGroup(tableNameB);
assertTrue(tableGrpB.getName().equals(RSGroupInfo.DEFAULT_GROUP));
// change table's group
LOG.info("Moving table [" + tableNameA + "," + tableNameB + "] to " + newGroup.getName());
- rsGroupAdmin.moveTables(Sets.newHashSet(tableNameA, tableNameB), newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableNameA, tableNameB), newGroup.getName());
// verify group change
Assert.assertEquals(newGroup.getName(),
- rsGroupAdmin.getRSGroupInfoOfTable(tableNameA).getName());
+ rsGroupAdmin.getRSGroup(tableNameA).getName());
Assert.assertEquals(newGroup.getName(),
- rsGroupAdmin.getRSGroupInfoOfTable(tableNameB).getName());
+ rsGroupAdmin.getRSGroup(tableNameB).getName());
// verify tables' not exist in old group
Set<TableName> DefaultTables =
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getTables();
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getTables();
assertFalse(DefaultTables.contains(tableNameA));
assertFalse(DefaultTables.contains(tableNameB));
// verify tables' exist in new group
- Set<TableName> newGroupTables = rsGroupAdmin.getRSGroupInfo(newGroupName).getTables();
+ Set<TableName> newGroupTables = rsGroupAdmin.getRSGroup(newGroupName).getTables();
assertTrue(newGroupTables.contains(tableNameA));
assertTrue(newGroupTables.contains(tableNameB));
}
@@ -315,7 +324,7 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
@Test
public void testTableMoveTruncateAndDrop() throws Exception {
final byte[] familyNameBytes = Bytes.toBytes("f");
- String newGroupName = getGroupName(name.getMethodName());
+ String newGroupName = getGroupName(getNameWithoutIndex(name.getMethodName()));
final RSGroupInfo newGroup = addGroup(newGroupName, 2);
TEST_UTIL.createMultiRegionTable(tableName, familyNameBytes, 5);
@@ -331,16 +340,17 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
});
- RSGroupInfo tableGrp = rsGroupAdmin.getRSGroupInfoOfTable(tableName);
+ RSGroupInfo tableGrp = rsGroupAdmin.getRSGroup(tableName);
+ LOG.info("got table group info is {}", tableGrp);
assertTrue(tableGrp.getName().equals(RSGroupInfo.DEFAULT_GROUP));
// change table's group
LOG.info("Moving table " + tableName + " to " + newGroup.getName());
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), newGroup.getName());
// verify group change
Assert.assertEquals(newGroup.getName(),
- rsGroupAdmin.getRSGroupInfoOfTable(tableName).getName());
+ rsGroupAdmin.getRSGroup(tableName).getName());
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
@@ -361,22 +371,20 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
// test truncate
admin.disableTable(tableName);
admin.truncateTable(tableName, true);
- Assert.assertEquals(1, rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getTables().size());
+ Assert.assertEquals(1, rsGroupAdmin.getRSGroup(newGroup.getName()).getTables().size());
Assert.assertEquals(tableName,
- rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getTables().first());
+ rsGroupAdmin.getRSGroup(newGroup.getName()).getTables().first());
// verify removed table is removed from group
TEST_UTIL.deleteTable(tableName);
- Assert.assertEquals(0, rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getTables().size());
+ Assert.assertEquals(0, rsGroupAdmin.getRSGroup(newGroup.getName()).getTables().size());
- assertTrue(observer.preMoveTablesCalled);
- assertTrue(observer.postMoveTablesCalled);
}
@Test
public void testDisabledTableMove() throws Exception {
final byte[] familyNameBytes = Bytes.toBytes("f");
- String newGroupName = getGroupName(name.getMethodName());
+ String newGroupName = getGroupName(getNameWithoutIndex(name.getMethodName()));
final RSGroupInfo newGroup = addGroup(newGroupName, 2);
TEST_UTIL.createMultiRegionTable(tableName, familyNameBytes, 5);
@@ -391,7 +399,7 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
});
- RSGroupInfo tableGrp = rsGroupAdmin.getRSGroupInfoOfTable(tableName);
+ RSGroupInfo tableGrp = rsGroupAdmin.getRSGroup(tableName);
assertTrue(tableGrp.getName().equals(RSGroupInfo.DEFAULT_GROUP));
// test disable table
@@ -399,18 +407,18 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
// change table's group
LOG.info("Moving table " + tableName + " to " + newGroup.getName());
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), newGroup.getName());
// verify group change
Assert.assertEquals(newGroup.getName(),
- rsGroupAdmin.getRSGroupInfoOfTable(tableName).getName());
+ rsGroupAdmin.getRSGroup(tableName).getName());
}
@Test
public void testNonExistentTableMove() throws Exception {
- TableName tableName = TableName.valueOf(tablePrefix + name.getMethodName());
-
- RSGroupInfo tableGrp = rsGroupAdmin.getRSGroupInfoOfTable(tableName);
+ TableName tableName = TableName.valueOf(tablePrefix +
+ getNameWithoutIndex(name.getMethodName()));
+ RSGroupInfo tableGrp = rsGroupAdmin.getRSGroup(tableName);
assertNull(tableGrp);
// test if table exists already.
@@ -419,21 +427,22 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
LOG.info("Moving table " + tableName + " to " + RSGroupInfo.DEFAULT_GROUP);
try {
- rsGroupAdmin.moveTables(Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
fail("Table " + tableName + " shouldn't have been successfully moved.");
} catch (IOException ex) {
assertTrue(ex instanceof TableNotFoundException);
}
try {
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(Address.fromParts("bogus", 123)),
- Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(Address.fromParts("bogus", 123)),
+ RSGroupInfo.DEFAULT_GROUP);
fail("Table " + tableName + " shouldn't have been successfully moved.");
} catch (IOException ex) {
assertTrue(ex instanceof TableNotFoundException);
}
// verify group change
- assertNull(rsGroupAdmin.getRSGroupInfoOfTable(tableName));
+ assertNull(rsGroupAdmin.getRSGroup(tableName));
}
@Test
@@ -512,20 +521,10 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
});
SortedSet<TableName> tables
- = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getTables();
+ = rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getTables();
assertTrue("Table 't1' must be in 'default' rsgroup", tables.contains(tn1));
// Cleanup
TEST_UTIL.deleteTable(tn1);
}
-
- private void toggleQuotaCheckAndRestartMiniCluster(boolean enable) throws Exception {
- TEST_UTIL.shutdownMiniCluster();
- TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, enable);
- TEST_UTIL.startMiniCluster(NUM_SLAVES_BASE - 1);
- TEST_UTIL.getConfiguration().setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART,
- NUM_SLAVES_BASE - 1);
- TEST_UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true);
- initialize();
- }
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
index 076362e..eb808e3 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hbase.rsgroup;
-import static org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.DEFAULT_MAX_RETRY_VALUE;
import static org.apache.hadoop.hbase.util.Threads.sleep;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@@ -55,12 +54,15 @@ import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
+@RunWith(Parameterized.class)
@Category({ LargeTests.class })
public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
@@ -128,7 +130,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
}
final ServerName targetServer = tmpTargetServer;
// move target server to group
- rsGroupAdmin.moveServers(Sets.newHashSet(targetServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(targetServer.getAddress()), newGroup.getName());
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
public boolean evaluate() throws Exception {
@@ -162,14 +164,14 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
int initNumGroups = rsGroupAdmin.listRSGroups().size();
RSGroupInfo appInfo = addGroup(getGroupName(name.getMethodName()), 1);
RSGroupInfo adminInfo = addGroup(getGroupName(name.getMethodName()), 1);
- RSGroupInfo dInfo = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo dInfo = rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
Assert.assertEquals(initNumGroups + 2, rsGroupAdmin.listRSGroups().size());
assertEquals(1, adminInfo.getServers().size());
assertEquals(1, appInfo.getServers().size());
assertEquals(getNumServers() - 2, dInfo.getServers().size());
- rsGroupAdmin.moveServers(appInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(appInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.removeRSGroup(appInfo.getName());
- rsGroupAdmin.moveServers(adminInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(adminInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.removeRSGroup(adminInfo.getName());
Assert.assertEquals(rsGroupAdmin.listRSGroups().size(), initNumGroups);
}
@@ -180,14 +182,14 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
addGroup("bar", 3);
rsGroupAdmin.addRSGroup("foo");
- RSGroupInfo barGroup = rsGroupAdmin.getRSGroupInfo("bar");
- RSGroupInfo fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ RSGroupInfo barGroup = rsGroupAdmin.getRSGroup("bar");
+ RSGroupInfo fooGroup = rsGroupAdmin.getRSGroup("foo");
assertEquals(3, barGroup.getServers().size());
assertEquals(0, fooGroup.getServers().size());
// test fail bogus server move
try {
- rsGroupAdmin.moveServers(Sets.newHashSet(Address.fromString("foo:9999")), "foo");
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(Address.fromString("foo:9999")), "foo");
fail("Bogus servers shouldn't have been successfully moved.");
} catch (IOException ex) {
String exp = "Source RSGroup for server foo:9999 does not exist.";
@@ -197,34 +199,34 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// test success case
LOG.info("moving servers " + barGroup.getServers() + " to group foo");
- rsGroupAdmin.moveServers(barGroup.getServers(), fooGroup.getName());
+ rsGroupAdmin.moveToRSGroup(barGroup.getServers(), fooGroup.getName());
- barGroup = rsGroupAdmin.getRSGroupInfo("bar");
- fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ barGroup = rsGroupAdmin.getRSGroup("bar");
+ fooGroup = rsGroupAdmin.getRSGroup("foo");
assertEquals(0, barGroup.getServers().size());
assertEquals(3, fooGroup.getServers().size());
LOG.info("moving servers " + fooGroup.getServers() + " to group default");
- rsGroupAdmin.moveServers(fooGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(fooGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
public boolean evaluate() throws Exception {
- return getNumServers() == rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP)
+ return getNumServers() == rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP)
.getServers().size();
}
});
- fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ fooGroup = rsGroupAdmin.getRSGroup("foo");
assertEquals(0, fooGroup.getServers().size());
// test group removal
LOG.info("Remove group " + barGroup.getName());
rsGroupAdmin.removeRSGroup(barGroup.getName());
- Assert.assertEquals(null, rsGroupAdmin.getRSGroupInfo(barGroup.getName()));
+ Assert.assertEquals(null, rsGroupAdmin.getRSGroup(barGroup.getName()));
LOG.info("Remove group " + fooGroup.getName());
rsGroupAdmin.removeRSGroup(fooGroup.getName());
- Assert.assertEquals(null, rsGroupAdmin.getRSGroupInfo(fooGroup.getName()));
+ Assert.assertEquals(null, rsGroupAdmin.getRSGroup(fooGroup.getName()));
}
@Test
@@ -236,7 +238,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// remove online servers
try {
- rsGroupAdmin.removeServers(Sets.newHashSet(targetServer.getAddress()));
+ rsGroupAdmin.removeRSGroup(Sets.newHashSet(targetServer.getAddress()));
fail("Online servers shouldn't have been successfully removed.");
} catch (IOException ex) {
String exp =
@@ -267,7 +269,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
});
try {
- rsGroupAdmin.removeServers(Sets.newHashSet(targetServer.getAddress()));
+ rsGroupAdmin.removeRSGroup(Sets.newHashSet(targetServer.getAddress()));
fail("Dead servers shouldn't have been successfully removed.");
} catch (IOException ex) {
String exp = "Server " + targetServer.getAddress() + " is on the dead servers list," +
@@ -287,8 +289,8 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
assertEquals(1, admin.listDecommissionedRegionServers().size());
assertTrue(newGroup.getServers().contains(targetServer.getAddress()));
- rsGroupAdmin.removeServers(Sets.newHashSet(targetServer.getAddress()));
- Set<Address> newGroupServers = rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getServers();
+ rsGroupAdmin.removeRSGroup(Sets.newHashSet(targetServer.getAddress()));
+ Set<Address> newGroupServers = rsGroupAdmin.getRSGroup(newGroup.getName()).getServers();
assertFalse(newGroupServers.contains(targetServer.getAddress()));
assertEquals(2, newGroupServers.size());
@@ -320,7 +322,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
for (ServerName server : admin.getClusterMetrics(EnumSet.of(Option.LIVE_SERVERS))
.getLiveServerMetrics().keySet()) {
if (!newGroup.containsServer(server.getAddress()) &&
- !rsGroupAdmin.getRSGroupInfo("master").containsServer(server.getAddress())) {
+ !rsGroupAdmin.getRSGroup("master").containsServer(server.getAddress())) {
targetServer = server;
break;
}
@@ -328,14 +330,15 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
LOG.debug("Print group info : " + rsGroupAdmin.listRSGroups());
int oldDefaultGroupServerSize =
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().size();
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().size();
int oldDefaultGroupTableSize =
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getTables().size();
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getTables().size();
// test fail bogus server move
try {
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(Address.fromString("foo:9999")),
- Sets.newHashSet(tableName), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(Address.fromString("foo:9999")),
+ newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), newGroup.getName());
fail("Bogus servers shouldn't have been successfully moved.");
} catch (IOException ex) {
String exp = "Source RSGroup for server foo:9999 does not exist.";
@@ -344,18 +347,19 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
}
// test move when src = dst
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(targetServer.getAddress()),
- Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(targetServer.getAddress()),
+ RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), RSGroupInfo.DEFAULT_GROUP);
// verify default group info
Assert.assertEquals(oldDefaultGroupServerSize,
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().size());
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().size());
Assert.assertEquals(oldDefaultGroupTableSize,
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getTables().size());
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getTables().size());
// verify new group info
- Assert.assertEquals(1, rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getServers().size());
- Assert.assertEquals(0, rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getTables().size());
+ Assert.assertEquals(1, rsGroupAdmin.getRSGroup(newGroup.getName()).getServers().size());
+ Assert.assertEquals(0, rsGroupAdmin.getRSGroup(newGroup.getName()).getTables().size());
// get all region to move targetServer
List<String> regionList = getTableRegionMap().get(tableName);
@@ -381,38 +385,39 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// move targetServer and table to newGroup
LOG.info("moving server and table to newGroup");
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(targetServer.getAddress()),
- Sets.newHashSet(tableName), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(targetServer.getAddress()),
+ newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(tableName), newGroup.getName());
// verify group change
Assert.assertEquals(newGroup.getName(),
- rsGroupAdmin.getRSGroupInfoOfTable(tableName).getName());
+ rsGroupAdmin.getRSGroup(tableName).getName());
// verify servers' not exist in old group
Set<Address> defaultServers =
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers();
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers();
assertFalse(defaultServers.contains(targetServer.getAddress()));
// verify servers' exist in new group
- Set<Address> newGroupServers = rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getServers();
+ Set<Address> newGroupServers = rsGroupAdmin.getRSGroup(newGroup.getName()).getServers();
assertTrue(newGroupServers.contains(targetServer.getAddress()));
// verify tables' not exist in old group
Set<TableName> defaultTables =
- rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getTables();
+ rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getTables();
assertFalse(defaultTables.contains(tableName));
// verify tables' exist in new group
- Set<TableName> newGroupTables = rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getTables();
+ Set<TableName> newGroupTables = rsGroupAdmin.getRSGroup(newGroup.getName()).getTables();
assertTrue(newGroupTables.contains(tableName));
// verify that all region still assgin on targetServer
// TODO: uncomment after we reimplement moveServersAndTables, now the implementation is
- // moveServers first and then moveTables, so the region will be moved to other region servers.
+ // moveToRSGroup first and then moveTables, so the region will be moved to other region servers.
// Assert.assertEquals(5, getTableServerRegionMap().get(tableName).get(targetServer).size());
- assertTrue(observer.preMoveServersAndTables);
- assertTrue(observer.postMoveServersAndTables);
+ assertTrue(observer.preMoveServersCalled);
+ assertTrue(observer.postMoveServersCalled);
}
@Test
@@ -420,17 +425,17 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// create groups and assign servers
rsGroupAdmin.addRSGroup("foo");
- RSGroupInfo fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ RSGroupInfo fooGroup = rsGroupAdmin.getRSGroup("foo");
assertEquals(0, fooGroup.getServers().size());
- RSGroupInfo defaultGroup = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo defaultGroup = rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
// test remove all servers from default
try {
- rsGroupAdmin.moveServers(defaultGroup.getServers(), fooGroup.getName());
- fail(RSGroupAdminServer.KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE);
+ rsGroupAdmin.moveToRSGroup(defaultGroup.getServers(), fooGroup.getName());
+ fail(RSGroupInfoManagerImpl.KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE);
} catch (ConstraintException ex) {
assertTrue(
- ex.getMessage().contains(RSGroupAdminServer.KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE));
+ ex.getMessage().contains(RSGroupInfoManagerImpl.KEEP_ONE_SERVER_IN_DEFAULT_ERROR_MESSAGE));
}
// test success case, remove one server from default ,keep at least one server
@@ -438,35 +443,35 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
Address serverInDefaultGroup = defaultGroup.getServers().iterator().next();
LOG.info("moving server " + serverInDefaultGroup + " from group default to group " +
fooGroup.getName());
- rsGroupAdmin.moveServers(Sets.newHashSet(serverInDefaultGroup), fooGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(serverInDefaultGroup), fooGroup.getName());
}
- fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ fooGroup = rsGroupAdmin.getRSGroup("foo");
LOG.info("moving servers " + fooGroup.getServers() + " to group default");
- rsGroupAdmin.moveServers(fooGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(fooGroup.getServers(), RSGroupInfo.DEFAULT_GROUP);
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
public boolean evaluate() throws Exception {
- return getNumServers() == rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP)
+ return getNumServers() == rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP)
.getServers().size();
}
});
- fooGroup = rsGroupAdmin.getRSGroupInfo("foo");
+ fooGroup = rsGroupAdmin.getRSGroup("foo");
assertEquals(0, fooGroup.getServers().size());
// test group removal
LOG.info("Remove group " + fooGroup.getName());
rsGroupAdmin.removeRSGroup(fooGroup.getName());
- Assert.assertEquals(null, rsGroupAdmin.getRSGroupInfo(fooGroup.getName()));
+ Assert.assertEquals(null, rsGroupAdmin.getRSGroup(fooGroup.getName()));
}
@Test
public void testFailedMoveBeforeRetryExhaustedWhenMoveServer() throws Exception {
String groupName = getGroupName(name.getMethodName());
rsGroupAdmin.addRSGroup(groupName);
- final RSGroupInfo newGroup = rsGroupAdmin.getRSGroupInfo(groupName);
+ final RSGroupInfo newGroup = rsGroupAdmin.getRSGroup(groupName);
Pair<ServerName, RegionStateNode> gotPair = createTableWithRegionSplitting(newGroup,
10);
@@ -482,7 +487,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
Thread t2 = new Thread(() -> {
LOG.info("thread2 start running, to move regions");
try {
- rsGroupAdmin.moveServers(Sets.newHashSet(movedServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(movedServer.getAddress()), newGroup.getName());
} catch (IOException e) {
LOG.error("move server error", e);
}
@@ -512,7 +517,8 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// wait until there is only left the region we changed state and recover its state.
// wait time is set according to the number of max retries, all except failed regions will be
// moved in one retry, and will sleep 1s until next retry.
- while (System.currentTimeMillis() - current <= DEFAULT_MAX_RETRY_VALUE * 1000) {
+ while (System.currentTimeMillis() - current <=
+ RSGroupInfoManagerImpl.DEFAULT_MAX_RETRY_VALUE * 1000) {
List<RegionInfo> regions = getRegions.apply(owner);
LOG.debug("server table region size is:{}", regions.size());
assert regions.size() >= 1;
@@ -600,7 +606,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
@Test
public void testFailedMoveServersAndRepair() throws Exception{
- // This UT calls moveServers() twice to test the idempotency of it.
+ // This UT calls moveToRSGroup() twice to test the idempotency of it.
// The first time, movement fails because a region is made in SPLITTING state
// which will not be moved.
// The second time, the region state is OPEN and check if all
@@ -616,7 +622,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// move server to newGroup and check regions
try {
- rsGroupAdmin.moveServers(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
fail("should get IOException when retry exhausted but there still exists failed moved "
+ "regions");
}catch (Exception e){
@@ -632,7 +638,7 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// retry move server to newGroup and check if all regions on srcServer was moved
rsn.setState(RegionState.State.OPEN);
- rsGroupAdmin.moveServers(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
assertEquals(master.getAssignmentManager().getRegionsOnServer(srcServer).size(), 0);
}
@@ -661,8 +667,8 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// move server and table to newGroup and check regions
try {
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(srcServer.getAddress()),
- Sets.newHashSet(table2), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(table2), newGroup.getName());
fail("should get IOException when retry exhausted but there still exists failed moved "
+ "regions");
}catch (Exception e){
@@ -679,8 +685,8 @@ public class TestRSGroupsAdmin2 extends TestRSGroupsBase {
// retry moveServersAndTables to newGroup and check if all regions on srcServer belongs to
// table2
rsn.setState(RegionState.State.OPEN);
- rsGroupAdmin.moveServersAndTables(Sets.newHashSet(srcServer.getAddress()),
- Sets.newHashSet(table2), newGroup.getName());
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(srcServer.getAddress()), newGroup.getName());
+ rsGroupAdmin.setRSGroup(Sets.newHashSet(table2), newGroup.getName());
for(RegionInfo regionsInfo : master.getAssignmentManager().getRegionsOnServer(srcServer)){
assertEquals(regionsInfo.getTable(), table2);
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
index 8d10850..be93186 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
@@ -42,9 +42,12 @@ import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+@RunWith(Parameterized.class)
@Category({ MediumTests.class })
public class TestRSGroupsBalance extends TestRSGroupsBase {
@@ -80,7 +83,8 @@ public class TestRSGroupsBalance extends TestRSGroupsBase {
String newGroupName = getGroupName(name.getMethodName());
addGroup(newGroupName, 3);
- final TableName tableName = TableName.valueOf(tablePrefix + "_ns", name.getMethodName());
+ final TableName tableName = TableName.valueOf(tablePrefix + "_ns",
+ getNameWithoutIndex(name.getMethodName()));
admin.createNamespace(NamespaceDescriptor.create(tableName.getNamespaceAsString())
.addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, newGroupName).build());
final TableDescriptor desc = TableDescriptorBuilder.newBuilder(tableName)
@@ -151,13 +155,12 @@ public class TestRSGroupsBalance extends TestRSGroupsBase {
@Test
public void testMisplacedRegions() throws Exception {
- String namespace = tablePrefix + "_" + name.getMethodName();
+ String namespace = tablePrefix + "_" + getNameWithoutIndex(name.getMethodName());
TEST_UTIL.getAdmin().createNamespace(NamespaceDescriptor.create(namespace).build());
- final TableName tableName =
- TableName.valueOf(namespace, tablePrefix + "_" + name.getMethodName());
- LOG.info(name.getMethodName());
+ final TableName tableName = TableName.valueOf(namespace, tablePrefix + "_" +
+ getNameWithoutIndex(name.getMethodName()));
- final RSGroupInfo rsGroupInfo = addGroup(name.getMethodName(), 1);
+ final RSGroupInfo rsGroupInfo = addGroup(getGroupName(name.getMethodName()), 1);
TEST_UTIL.createMultiRegionTable(tableName, new byte[] { 'f' }, 15);
TEST_UTIL.waitUntilAllRegionsAssigned(tableName);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
index b9885be..6626fba 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
@@ -21,6 +21,7 @@ import static org.junit.Assert.assertTrue;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.LinkedList;
@@ -30,6 +31,7 @@ import java.util.Optional;
import java.util.Random;
import java.util.Set;
import java.util.TreeMap;
+import java.util.function.Supplier;
import java.util.regex.Pattern;
import org.apache.hadoop.hbase.ClusterMetrics;
import org.apache.hadoop.hbase.ClusterMetrics.Option;
@@ -54,8 +56,10 @@ import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
import org.apache.hadoop.hbase.master.ServerManager;
import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
import org.apache.hadoop.hbase.net.Address;
+import org.apache.hadoop.hbase.quotas.QuotaUtil;
import org.junit.Rule;
import org.junit.rules.TestName;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -74,7 +78,6 @@ public abstract class TestRSGroupsBase {
protected static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
protected static Admin admin;
protected static HBaseCluster cluster;
- protected static RSGroupAdminClient rsGroupAdmin;
protected static HMaster master;
protected boolean INIT = false;
protected static RSGroupAdminEndpoint rsGroupAdminEndpoint;
@@ -89,6 +92,48 @@ public abstract class TestRSGroupsBase {
public TestName name = new TestName();
protected TableName tableName;
+ protected Admin rsGroupAdmin;
+
+ @Parameterized.Parameter
+ public Supplier<Object> getAdmin;
+
+ private static RSGroupAdminClient getRSGroupAdmin(){
+ try {
+ return new VerifyingRSGroupAdminClient(
+ new RSGroupAdminClient(TEST_UTIL.getConnection()), TEST_UTIL.getConfiguration());
+ } catch (IOException e) {
+ LOG.error("Get group admin failed", e);
+ return null;
+ }
+ }
+
+ private static Admin getAdmin(){
+ try {
+ return TEST_UTIL.getAdmin();
+ } catch (IOException e) {
+ LOG.error("Get hbase admin failed", e);
+ return null;
+ }
+ }
+
+ public static Object resetAdminConnection(Object admin) {
+ if(admin instanceof RSGroupAdminClient) {
+ return getRSGroupAdmin();
+ }else {
+ return getAdmin();
+ }
+ }
+
+ public static String getNameWithoutIndex(String name) {
+ return name.split("\\[")[0];
+ }
+
+ @Parameterized.Parameters
+ public static List<Object[]> params() {
+ return Arrays.asList(new Supplier<?>[] { TestRSGroupsBase::getRSGroupAdmin },
+ new Supplier<?>[] { TestRSGroupsBase::getAdmin });
+ }
+
public static void setUpTestBeforeClass() throws Exception {
TEST_UTIL.getConfiguration().setFloat(
"hbase.master.balancer.stochastic.tableSkewCost", 6000);
@@ -107,6 +152,10 @@ public abstract class TestRSGroupsBase {
initialize();
}
+ public void setAdmin(){
+ rsGroupAdmin = (Admin) getAdmin.get();
+ }
+
protected static void initialize() throws Exception {
admin = TEST_UTIL.getAdmin();
cluster = TEST_UTIL.getHBaseCluster();
@@ -121,8 +170,6 @@ public abstract class TestRSGroupsBase {
}
});
admin.balancerSwitch(false, true);
- rsGroupAdmin = new VerifyingRSGroupAdminClient(
- new RSGroupAdminClient(TEST_UTIL.getConnection()), TEST_UTIL.getConfiguration());
MasterCoprocessorHost host = master.getMasterCoprocessorHost();
observer = (CPMasterObserver) host.findCoprocessor(CPMasterObserver.class.getName());
rsGroupAdminEndpoint = (RSGroupAdminEndpoint)
@@ -134,8 +181,9 @@ public abstract class TestRSGroupsBase {
}
public void setUpBeforeMethod() throws Exception {
+ setAdmin();
LOG.info(name.getMethodName());
- tableName = TableName.valueOf(tablePrefix + "_" + name.getMethodName());
+ tableName = TableName.valueOf(tablePrefix + "_" + name.getMethodName().split("\\[")[0]);
if (!INIT) {
INIT = true;
tearDownAfterMethod();
@@ -164,7 +212,8 @@ public abstract class TestRSGroupsBase {
((MiniHBaseCluster)cluster).getMaster().getServerName();
try {
- rsGroupAdmin.moveServers(Sets.newHashSet(masterServerName.getAddress()), "master");
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(masterServerName.getAddress()),
+ "master");
} catch (Exception ex) {
LOG.warn("Got this on setup, FYI", ex);
}
@@ -176,7 +225,7 @@ public abstract class TestRSGroupsBase {
//Might be greater since moving servers back to default
//is after starting a server
- return rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().size()
+ return rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().size()
== NUM_SLAVES_BASE;
}
});
@@ -184,7 +233,7 @@ public abstract class TestRSGroupsBase {
protected RSGroupInfo addGroup(String groupName, int serverCount)
throws IOException, InterruptedException {
- RSGroupInfo defaultInfo = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo defaultInfo = rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.addRSGroup(groupName);
Set<Address> set = new HashSet<>();
for (Address server : defaultInfo.getServers()) {
@@ -193,15 +242,15 @@ public abstract class TestRSGroupsBase {
}
set.add(server);
}
- rsGroupAdmin.moveServers(set, groupName);
- RSGroupInfo result = rsGroupAdmin.getRSGroupInfo(groupName);
+ rsGroupAdmin.moveToRSGroup(set, groupName);
+ RSGroupInfo result = rsGroupAdmin.getRSGroup(groupName);
return result;
}
- protected void removeGroup(String groupName) throws IOException {
- RSGroupInfo groupInfo = rsGroupAdmin.getRSGroupInfo(groupName);
- rsGroupAdmin.moveTables(groupInfo.getTables(), RSGroupInfo.DEFAULT_GROUP);
- rsGroupAdmin.moveServers(groupInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
+ public void removeGroup(String groupName) throws IOException {
+ RSGroupInfo groupInfo = rsGroupAdmin.getRSGroup(groupName);
+ rsGroupAdmin.setRSGroup(groupInfo.getTables(), RSGroupInfo.DEFAULT_GROUP);
+ rsGroupAdmin.moveToRSGroup(groupInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
rsGroupAdmin.removeRSGroup(groupName);
}
@@ -224,7 +273,7 @@ public abstract class TestRSGroupsBase {
RSGroupAdminClient groupAdmin = new RSGroupAdminClient(TEST_UTIL.getConnection());
for(RSGroupInfo group: groupAdmin.listRSGroups()) {
if(!group.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
- groupAdmin.moveTables(group.getTables(), RSGroupInfo.DEFAULT_GROUP);
+ groupAdmin.setRSGroup(group.getTables(), RSGroupInfo.DEFAULT_GROUP);
groupAdmin.moveServers(group.getServers(), RSGroupInfo.DEFAULT_GROUP);
groupAdmin.removeRSGroup(group.getName());
}
@@ -276,8 +325,9 @@ public abstract class TestRSGroupsBase {
return count;
}
- protected String getGroupName(String baseName) {
- return groupPrefix + "_" + baseName + "_" + rand.nextInt(Integer.MAX_VALUE);
+ public String getGroupName(String baseName) {
+ return groupPrefix + "_" + getNameWithoutIndex(baseName) + "_" +
+ rand.nextInt(Integer.MAX_VALUE);
}
/**
@@ -290,6 +340,17 @@ public abstract class TestRSGroupsBase {
.findFirst().get();
}
+ protected void toggleQuotaCheckAndRestartMiniCluster(boolean enable) throws Exception {
+ TEST_UTIL.shutdownMiniCluster();
+ TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, enable);
+ TEST_UTIL.startMiniCluster(NUM_SLAVES_BASE - 1);
+ TEST_UTIL.getConfiguration().setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART,
+ NUM_SLAVES_BASE - 1);
+ TEST_UTIL.getConfiguration().setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true);
+ initialize();
+ rsGroupAdmin = (Admin) resetAdminConnection(rsGroupAdmin);
+ }
+
public static class CPMasterObserver implements MasterCoprocessor, MasterObserver {
boolean preBalanceRSGroupCalled = false;
boolean postBalanceRSGroupCalled = false;
@@ -313,6 +374,8 @@ public abstract class TestRSGroupsBase {
boolean postListRSGroupsCalled = false;
boolean preGetRSGroupInfoOfServerCalled = false;
boolean postGetRSGroupInfoOfServerCalled = false;
+ boolean preSetRSGroupForTablesCalled = false;
+ boolean postSetRSGroupForTablesCalled = false;
public void resetFlags() {
preBalanceRSGroupCalled = false;
@@ -337,6 +400,8 @@ public abstract class TestRSGroupsBase {
postListRSGroupsCalled = false;
preGetRSGroupInfoOfServerCalled = false;
postGetRSGroupInfoOfServerCalled = false;
+ preSetRSGroupForTablesCalled = false;
+ postSetRSGroupForTablesCalled = false;
}
@Override
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
index 67ed2a9..45d2a2e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
@@ -34,7 +34,6 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaTableUtil;
-import org.apache.hadoop.hbase.quotas.QuotaUtil;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.After;
@@ -45,11 +44,14 @@ import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+@RunWith(Parameterized.class)
@Category({ MediumTests.class })
public class TestRSGroupsBasics extends TestRSGroupsBase {
@@ -81,10 +83,12 @@ public class TestRSGroupsBasics extends TestRSGroupsBase {
@Test
public void testBasicStartUp() throws IOException {
- RSGroupInfo defaultInfo = rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP);
+ RSGroupInfo defaultInfo = rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP);
assertEquals(NUM_SLAVES_BASE, defaultInfo.getServers().size());
// Assignment of meta and rsgroup regions.
int count = master.getAssignmentManager().getRegionStates().getRegionAssignments().size();
+ LOG.info("regions assignments are" +
+ master.getAssignmentManager().getRegionStates().getRegionAssignments().toString());
// 2 (meta and rsgroup)
assertEquals(2, count);
}
@@ -165,6 +169,7 @@ public class TestRSGroupsBasics extends TestRSGroupsBase {
// clone
admin.cloneSnapshot(snapshotName, clonedTableName);
+ admin.deleteSnapshot(snapshotName);
}
@Test
@@ -206,7 +211,7 @@ public class TestRSGroupsBasics extends TestRSGroupsBase {
assertEquals(0, notClearedServers.size());
// the stopped region server gets cleared and removed from the group
- Set<Address> newGroupServers = rsGroupAdmin.getRSGroupInfo(newGroup.getName()).getServers();
+ Set<Address> newGroupServers = rsGroupAdmin.getRSGroup(newGroup.getName()).getServers();
assertFalse(newGroupServers.contains(serverToStop.getAddress()));
assertEquals(serverCountToMoveToNewGroup - 1 /* 1 stopped */, newGroupServers.size());
}
@@ -244,34 +249,20 @@ public class TestRSGroupsBasics extends TestRSGroupsBase {
assertEquals(serverCountToMoveToDeadServerGroup, notClearedServers.size());
Set<Address> ServersInDeadServerGroup =
- rsGroupAdmin.getRSGroupInfo(deadServerGroup.getName()).getServers();
+ rsGroupAdmin.getRSGroup(deadServerGroup.getName()).getServers();
assertEquals(serverCountToMoveToDeadServerGroup, ServersInDeadServerGroup.size());
assertTrue(ServersInDeadServerGroup.contains(serverToStop.getAddress()));
}
@Test
public void testRSGroupsWithHBaseQuota() throws Exception {
- TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, true);
- restartHBaseCluster();
- try {
- TEST_UTIL.waitFor(90000, new Waiter.Predicate<Exception>() {
- @Override
- public boolean evaluate() throws Exception {
- return admin.isTableAvailable(QuotaTableUtil.QUOTA_TABLE_NAME);
- }
- });
- } finally {
- TEST_UTIL.getConfiguration().setBoolean(QuotaUtil.QUOTA_CONF_KEY, false);
- restartHBaseCluster();
- }
- }
-
- private void restartHBaseCluster() throws Exception {
- LOG.info("\n\nShutting down cluster");
- TEST_UTIL.shutdownMiniHBaseCluster();
- LOG.info("\n\nSleeping a bit");
- Thread.sleep(2000);
- TEST_UTIL.restartHBaseCluster(NUM_SLAVES_BASE - 1);
- initialize();
+ toggleQuotaCheckAndRestartMiniCluster(true);
+ TEST_UTIL.waitFor(90000, new Waiter.Predicate<Exception>() {
+ @Override
+ public boolean evaluate() throws Exception {
+ return admin.isTableAvailable(QuotaTableUtil.QUOTA_TABLE_NAME);
+ }
+ });
+ toggleQuotaCheckAndRestartMiniCluster(false);
}
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
index 539f351..2593216 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
@@ -51,11 +51,14 @@ import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
+@RunWith(Parameterized.class)
@Category({ MediumTests.class })
public class TestRSGroupsKillRS extends TestRSGroupsBase {
@@ -88,7 +91,8 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
@Test
public void testKillRS() throws Exception {
RSGroupInfo appInfo = addGroup("appInfo", 1);
- final TableName tableName = TableName.valueOf(tablePrefix + "_ns", name.getMethodName());
+ final TableName tableName = TableName.valueOf(tablePrefix + "_ns",
+ getNameWithoutIndex(name.getMethodName()));
admin.createNamespace(NamespaceDescriptor.create(tableName.getNamespaceAsString())
.addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, appInfo.getName()).build());
final TableDescriptor desc = TableDescriptorBuilder.newBuilder(tableName)
@@ -128,8 +132,8 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
});
Set<Address> newServers = Sets.newHashSet();
newServers
- .add(rsGroupAdmin.getRSGroupInfo(RSGroupInfo.DEFAULT_GROUP).getServers().iterator().next());
- rsGroupAdmin.moveServers(newServers, appInfo.getName());
+ .add(rsGroupAdmin.getRSGroup(RSGroupInfo.DEFAULT_GROUP).getServers().iterator().next());
+ rsGroupAdmin.moveToRSGroup(newServers, appInfo.getName());
// Make sure all the table's regions get reassigned
// disabling the table guarantees no conflicting assign/unassign (ie SSH) happens
@@ -161,12 +165,12 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
TEST_UTIL.loadTable(t, Bytes.toBytes("f"));
Set<TableName> toAddTables = new HashSet<>();
toAddTables.add(tableName);
- rsGroupAdmin.moveTables(toAddTables, groupName);
- assertTrue(rsGroupAdmin.getRSGroupInfo(groupName).getTables().contains(tableName));
+ rsGroupAdmin.setRSGroup(toAddTables, groupName);
+ assertTrue(rsGroupAdmin.getRSGroup(groupName).getTables().contains(tableName));
TEST_UTIL.waitTableAvailable(tableName, 30000);
// check my_group servers and table regions
- Set<Address> servers = rsGroupAdmin.getRSGroupInfo(groupName).getServers();
+ Set<Address> servers = rsGroupAdmin.getRSGroup(groupName).getServers();
assertEquals(2, servers.size());
LOG.debug("group servers {}", servers);
for (RegionInfo tr :
@@ -197,7 +201,7 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
// case 1: stop all the regionservers in my_group, and restart a regionserver in my_group,
// and then check if all table regions are online
- for(Address addr : rsGroupAdmin.getRSGroupInfo(groupName).getServers()) {
+ for(Address addr : rsGroupAdmin.getRSGroup(groupName).getServers()) {
TEST_UTIL.getMiniHBaseCluster().stopRegionServer(getServerName(addr));
}
// better wait for a while for region reassign
@@ -224,7 +228,7 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
assertEquals(NUM_SLAVES_BASE - gsn.size(),
TEST_UTIL.getMiniHBaseCluster().getLiveRegionServerThreads().size());
ServerName newServer = master.getServerManager().getOnlineServersList().get(0);
- rsGroupAdmin.moveServers(Sets.newHashSet(newServer.getAddress()), groupName);
+ rsGroupAdmin.moveToRSGroup(Sets.newHashSet(newServer.getAddress()), groupName);
// wait and check if table regions are online
TEST_UTIL.waitTableAvailable(tableName, 30000);
}
@@ -240,14 +244,14 @@ public class TestRSGroupsKillRS extends TestRSGroupsBase {
tableName = TableName.META_TABLE_NAME;
Set<TableName> toAddTables = new HashSet<>();
toAddTables.add(tableName);
- rsGroupAdmin.moveTables(toAddTables, groupName);
- assertTrue(rsGroupAdmin.getRSGroupInfo(groupName).getTables().contains(tableName));
+ rsGroupAdmin.setRSGroup(toAddTables, groupName);
+ assertTrue(rsGroupAdmin.getRSGroup(groupName).getTables().contains(tableName));
TEST_UTIL.waitTableAvailable(tableName, 30000);
// restart the regionserver in meta_group, and lower its version
String originVersion = "";
Set<Address> servers = new HashSet<>();
- for(Address addr : rsGroupAdmin.getRSGroupInfo(groupName).getServers()) {
+ for(Address addr : rsGroupAdmin.getRSGroup(groupName).getServers()) {
servers.add(addr);
TEST_UTIL.getMiniHBaseCluster().stopRegionServer(getServerName(addr));
originVersion = master.getRegionServerVersion(getServerName(addr));
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
index 7ac1a49..cfc721f 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
@@ -56,7 +56,7 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
// online. In new master, RSGroupInfoManagerImpl gets the data from zk and waits for the expected
// assignment with a timeout.
@Category(MediumTests.class)
-public class TestRSGroupsOfflineMode {
+public class TestRSGroupsOfflineMode extends TestRSGroupsBase {
@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
@@ -106,7 +106,7 @@ public class TestRSGroupsOfflineMode {
@Test
public void testOffline() throws Exception, InterruptedException {
// Table should be after group table name so it gets assigned later.
- final TableName failoverTable = TableName.valueOf(name.getMethodName());
+ final TableName failoverTable = TableName.valueOf(getNameWithoutIndex(name.getMethodName()));
TEST_UTIL.createTable(failoverTable, Bytes.toBytes("f"));
final HRegionServer killRS = ((MiniHBaseCluster) cluster).getRegionServer(0);
final HRegionServer groupRS = ((MiniHBaseCluster) cluster).getRegionServer(1);
@@ -139,7 +139,7 @@ public class TestRSGroupsOfflineMode {
}
});
// Move table to group and wait.
- groupAdmin.moveTables(Sets.newHashSet(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME), newGroup);
+ groupAdmin.setRSGroup(Sets.newHashSet(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME), newGroup);
LOG.info("Waiting for move table...");
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
index c1c157a..06c4e4e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
@@ -21,6 +21,8 @@ import static org.apache.hadoop.hbase.AuthUtil.toGroupEntry;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
+import java.io.IOException;
+import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtility;
@@ -31,7 +33,11 @@ import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
+import org.apache.hadoop.hbase.ipc.RpcServer;
+import org.apache.hadoop.hbase.master.HMaster;
import org.apache.hadoop.hbase.security.User;
+import org.apache.hadoop.hbase.security.UserProvider;
+import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.hadoop.hbase.security.access.AccessControlClient;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.PermissionStorage;
@@ -92,6 +98,9 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
private static byte[] TEST_FAMILY = Bytes.toBytes("f1");
private static RSGroupAdminEndpoint rsGroupAdminEndpoint;
+ private static HMaster master;
+ private static AccessChecker accessChecker;
+ private static UserProvider userProvider;
@BeforeClass
public static void setupBeforeClass() throws Exception {
@@ -131,6 +140,22 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
systemUserConnection = TEST_UTIL.getConnection();
setUpTableAndUserPermissions();
+ master = TEST_UTIL.getHBaseCluster().getMaster();
+ accessChecker = master.getAccessChecker();
+ userProvider = UserProvider.instantiate(TEST_UTIL.getConfiguration());
+ }
+
+ private void checkPermission(String request) throws IOException {
+ accessChecker.requirePermission(getActiveUser(), request, null, Permission.Action.ADMIN);
+ }
+
+ private User getActiveUser() throws IOException {
+ // for non-rpc handling, fallback to system user
+ Optional<User> optionalUser = RpcServer.getRequestUser();
+ if (optionalUser.isPresent()) {
+ return optionalUser.get();
+ }
+ return userProvider.getCurrent();
}
private static void setUpTableAndUserPermissions() throws Exception {
@@ -204,7 +229,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testGetRSGroupInfo() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfo");
+ checkPermission("getRSGroupInfo");
return null;
};
@@ -214,7 +239,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testGetRSGroupInfoOfTable() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfoOfTable");
+ checkPermission("getRSGroupInfoOfTable");
return null;
};
@@ -224,7 +249,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testMoveServers() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveServers");
+ checkPermission("moveServers");
return null;
};
@@ -234,7 +259,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testMoveTables() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveTables");
+ checkPermission("moveTables");
return null;
};
@@ -244,7 +269,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testAddRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("addRSGroup");
+ checkPermission("addRSGroup");
return null;
};
@@ -254,7 +279,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testRemoveRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("removeRSGroup");
+ checkPermission("removeRSGroup");
return null;
};
@@ -264,7 +289,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testBalanceRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("balanceRSGroup");
+ checkPermission("balanceRSGroup");
return null;
};
@@ -274,7 +299,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testListRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("listRSGroup");
+ checkPermission("listRSGroup");
return null;
};
@@ -284,7 +309,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testGetRSGroupInfoOfServer() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfoOfServer");
+ checkPermission("getRSGroupInfoOfServer");
return null;
};
@@ -294,7 +319,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testMoveServersAndTables() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveServersAndTables");
+ checkPermission("moveServersAndTables");
return null;
};
@@ -304,7 +329,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil {
@Test
public void testRemoveServers() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.getGroupAdminService().checkPermission("removeServers");
+ checkPermission("removeServers");
return null;
};
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
index 28131a9..b189697 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
@@ -73,24 +73,18 @@ public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
}
@Override
- public RSGroupInfo getRSGroupInfo(String groupName) throws IOException {
+ public RSGroupInfo getRSGroup(String groupName) throws IOException {
return wrapped.getRSGroupInfo(groupName);
}
@Override
- public RSGroupInfo getRSGroupInfoOfTable(TableName tableName) throws IOException {
- return wrapped.getRSGroupInfoOfTable(tableName);
+ public RSGroupInfo getRSGroup(TableName tableName) throws IOException {
+ return wrapped.getRSGroup(tableName);
}
@Override
- public void moveServers(Set<Address> servers, String targetGroup) throws IOException {
- wrapped.moveServers(servers, targetGroup);
- verify();
- }
-
- @Override
- public void moveTables(Set<TableName> tables, String targetGroup) throws IOException {
- wrapped.moveTables(tables, targetGroup);
+ public void moveToRSGroup(Set<Address> servers, String targetGroup) throws IOException {
+ wrapped.moveToRSGroup(servers, targetGroup);
verify();
}
@@ -111,20 +105,19 @@ public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
}
@Override
- public RSGroupInfo getRSGroupOfServer(Address hostPort) throws IOException {
- return wrapped.getRSGroupOfServer(hostPort);
+ public RSGroupInfo getRSGroup(Address hostPort) throws IOException {
+ return wrapped.getRSGroup(hostPort);
}
@Override
- public void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String targetGroup)
- throws IOException {
- wrapped.moveServersAndTables(servers, tables, targetGroup);
+ public void removeRSGroup(Set<Address> servers) throws IOException {
+ wrapped.removeRSGroup(servers);
verify();
}
@Override
- public void removeServers(Set<Address> servers) throws IOException {
- wrapped.removeServers(servers);
+ public void setRSGroup(Set<TableName> tables, String groupName) throws IOException{
+ wrapped.setRSGroup(tables, groupName);
verify();
}
diff --git a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
index f3fdde7..52d032e 100644
--- a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
+++ b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
@@ -50,12 +50,14 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.replication.TableCFs;
import org.apache.hadoop.hbase.client.security.SecurityCapability;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.quotas.QuotaFilter;
import org.apache.hadoop.hbase.quotas.QuotaSettings;
import org.apache.hadoop.hbase.quotas.SpaceQuotaSnapshot;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest;
import org.apache.hadoop.hbase.security.access.Permission;
import org.apache.hadoop.hbase.security.access.UserPermission;
@@ -1158,6 +1160,56 @@ public class ThriftAdmin implements Admin {
}
@Override
+ public RSGroupInfo getRSGroup(String groupName) {
+ throw new NotImplementedException("getRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public void moveToRSGroup(Set<Address> servers, String targetGroup) {
+ throw new NotImplementedException("moveToRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public void addRSGroup(String groupName) {
+ throw new NotImplementedException("addRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public void removeRSGroup(String groupName) {
+ throw new NotImplementedException("removeRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public boolean balanceRSGroup(String groupName) {
+ throw new NotImplementedException("balanceRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public List<RSGroupInfo> listRSGroups() {
+ throw new NotImplementedException("listRSGroups not supported in ThriftAdmin");
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(Address hostPort) {
+ throw new NotImplementedException("getRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public void removeRSGroup(Set<Address> servers) {
+ throw new NotImplementedException("removeRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(TableName tableName) {
+ throw new NotImplementedException("getRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
+ public void setRSGroup(Set<TableName> tables, String groupName) {
+ throw new NotImplementedException("setRSGroup not supported in ThriftAdmin");
+ }
+
+ @Override
public Future<Void> splitRegionAsync(byte[] regionName) throws IOException {
return splitRegionAsync(regionName, null);
}
[hbase] 10/14: HBASE-22971 Deprecated RSGroupAdminEndpoint and make
RSGroup feature always enabled (#595)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 879b08c0b65e63b67e4abfc2b47ad88eaad06a95
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Tue Sep 24 12:09:12 2019 +0800
HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../hbase/favored/FavoredNodeLoadBalancer.java | 1 +
.../hadoop/hbase/favored/FavoredNodesPromoter.java | 2 +
.../org/apache/hadoop/hbase/master/HMaster.java | 43 +++----
.../apache/hadoop/hbase/master/LoadBalancer.java | 9 ++
.../hadoop/hbase/master/MasterRpcServices.java | 23 ++--
.../apache/hadoop/hbase/master/MasterServices.java | 2 +-
.../hbase/master/assignment/AssignmentManager.java | 15 ++-
.../master/balancer/FavoredStochasticBalancer.java | 1 +
.../hbase/master/balancer/LoadBalancerFactory.java | 18 ++-
.../AbstractStateMachineNamespaceProcedure.java | 11 ++
.../master/procedure/CreateNamespaceProcedure.java | 1 +
.../master/procedure/CreateTableProcedure.java | 27 +++-
.../master/procedure/MasterProcedureUtil.java | 50 +++++++-
.../master/procedure/ModifyNamespaceProcedure.java | 19 ++-
.../master/procedure/ModifyTableProcedure.java | 15 ++-
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 141 +--------------------
.../hbase/rsgroup/RSGroupBasedLoadBalancer.java | 65 ++++++----
.../hadoop/hbase/rsgroup/RSGroupableBalancer.java | 32 -----
.../org/apache/hadoop/hbase/TestNamespace.java | 2 +-
.../hadoop/hbase/master/AbstractTestDLS.java | 4 +-
.../hbase/master/MockNoopMasterServices.java | 2 +-
.../hadoop/hbase/master/TestClusterRestart.java | 6 +-
.../hadoop/hbase/master/TestMasterMetrics.java | 4 +-
.../TestMasterRestartAfterDisablingTable.java | 4 +-
.../hadoop/hbase/master/TestRollingRestart.java | 2 +-
.../TestFavoredStochasticBalancerPickers.java | 2 +-
.../hadoop/hbase/regionserver/TestRegionOpen.java | 2 +-
.../TestRegionReplicasWithRestartScenarios.java | 2 +-
.../regionserver/TestRegionServerMetrics.java | 4 +-
.../hbase/rsgroup/TestRSGroupsOfflineMode.java | 3 +-
.../security/access/TestTablePermissions.java | 3 +-
.../hbase/util/TestHBaseFsckReplication.java | 2 +
32 files changed, 241 insertions(+), 276 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeLoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeLoadBalancer.java
index b8d4b09..52a37a2 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeLoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeLoadBalancer.java
@@ -321,6 +321,7 @@ public class FavoredNodeLoadBalancer extends BaseLoadBalancer implements Favored
regionsOnServer.add(region);
}
+ @Override
public synchronized List<ServerName> getFavoredNodes(RegionInfo regionInfo) {
return this.fnm.getFavoredNodes(regionInfo);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesPromoter.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesPromoter.java
index 322eb1df..a24fce0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesPromoter.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodesPromoter.java
@@ -35,4 +35,6 @@ public interface FavoredNodesPromoter {
void generateFavoredNodesForMergedRegion(RegionInfo merged, RegionInfo [] mergeParents)
throws IOException;
+
+ List<ServerName> getFavoredNodes(RegionInfo regionInfo);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 3e0e18d..04c48a6 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -91,7 +91,6 @@ import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.executor.ExecutorType;
import org.apache.hadoop.hbase.favored.FavoredNodesManager;
-import org.apache.hadoop.hbase.favored.FavoredNodesPromoter;
import org.apache.hadoop.hbase.http.InfoServer;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
import org.apache.hadoop.hbase.ipc.RpcServer;
@@ -107,7 +106,6 @@ import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure;
import org.apache.hadoop.hbase.master.assignment.UnassignProcedure;
import org.apache.hadoop.hbase.master.balancer.BalancerChore;
-import org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer;
import org.apache.hadoop.hbase.master.balancer.ClusterStatusChore;
import org.apache.hadoop.hbase.master.balancer.LoadBalancerFactory;
import org.apache.hadoop.hbase.master.cleaner.DirScanPool;
@@ -188,6 +186,7 @@ import org.apache.hadoop.hbase.replication.master.ReplicationHFileCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationPeerConfigUpgrader;
import org.apache.hadoop.hbase.replication.regionserver.ReplicationStatus;
+import org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer;
import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.AccessDeniedException;
import org.apache.hadoop.hbase.security.SecurityConstants;
@@ -385,7 +384,7 @@ public class HMaster extends HRegionServer implements MasterServices {
private final LockManager lockManager = new LockManager(this);
- private LoadBalancer balancer;
+ private RSGroupBasedLoadBalancer balancer;
private RegionNormalizer normalizer;
private BalancerChore balancerChore;
private RegionNormalizerChore normalizerChore;
@@ -443,9 +442,6 @@ public class HMaster extends HRegionServer implements MasterServices {
private long splitPlanCount;
private long mergePlanCount;
- /* Handle favored nodes information */
- private FavoredNodesManager favoredNodesManager;
-
/** jetty server for master to redirect requests to regionserver infoServer */
private Server masterJettyServer;
@@ -774,7 +770,8 @@ public class HMaster extends HRegionServer implements MasterServices {
@VisibleForTesting
protected void initializeZKBasedSystemTrackers()
throws IOException, InterruptedException, KeeperException, ReplicationException {
- this.balancer = LoadBalancerFactory.getLoadBalancer(conf);
+ this.balancer = new RSGroupBasedLoadBalancer();
+ this.balancer.setConf(conf);
this.normalizer = RegionNormalizerFactory.getRegionNormalizer(conf);
this.normalizer.setMasterServices(this);
this.normalizer.setMasterRpcServices((MasterRpcServices)rpcServices);
@@ -1058,9 +1055,6 @@ public class HMaster extends HRegionServer implements MasterServices {
return temp;
});
}
- if (this.balancer instanceof FavoredNodesPromoter) {
- favoredNodesManager = new FavoredNodesManager(this);
- }
// initialize load balancer
this.balancer.setMasterServices(this);
@@ -1110,11 +1104,11 @@ public class HMaster extends HRegionServer implements MasterServices {
// table states messing up master launch (namespace table, etc., are not assigned).
this.assignmentManager.processOfflineRegions();
// Initialize after meta is up as below scans meta
- if (favoredNodesManager != null && !maintenanceMode) {
+ if (getFavoredNodesManager() != null && !maintenanceMode) {
SnapshotOfRegionAssignmentFromMeta snapshotOfRegionAssignment =
new SnapshotOfRegionAssignmentFromMeta(getConnection());
snapshotOfRegionAssignment.initialize();
- favoredNodesManager.initialize(snapshotOfRegionAssignment);
+ getFavoredNodesManager().initialize(snapshotOfRegionAssignment);
}
// set cluster status again after user regions are assigned
@@ -2064,14 +2058,13 @@ public class HMaster extends HRegionServer implements MasterServices {
LOG.debug("Unable to determine a plan to assign " + hri);
return;
}
- // TODO: What is this? I don't get it.
- if (dest.equals(serverName) && balancer instanceof BaseLoadBalancer
- && !((BaseLoadBalancer)balancer).shouldBeOnMaster(hri)) {
+ // TODO: deal with table on master for rs group.
+ if (dest.equals(serverName)) {
// To avoid unnecessary region moving later by balancer. Don't put user
// regions on master.
- LOG.debug("Skipping move of region " + hri.getRegionNameAsString()
- + " to avoid unnecessary region moving later by load balancer,"
- + " because it should not be on master");
+ LOG.debug("Skipping move of region " + hri.getRegionNameAsString() +
+ " to avoid unnecessary region moving later by load balancer," +
+ " because it should not be on master");
return;
}
}
@@ -3523,12 +3516,14 @@ public class HMaster extends HRegionServer implements MasterServices {
/**
* Fetch the configured {@link LoadBalancer} class name. If none is set, a default is returned.
- *
+ * <p/>
+ * Notice that, the base load balancer will always be {@link RSGroupBasedLoadBalancer} now, so
+ * this method will return the balancer used inside each rs group.
* @return The name of the {@link LoadBalancer} in use.
*/
public String getLoadBalancerClassName() {
- return conf.get(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, LoadBalancerFactory
- .getDefaultLoadBalancerClass().getName());
+ return conf.get(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
+ LoadBalancerFactory.getDefaultLoadBalancerClass().getName());
}
/**
@@ -3543,13 +3538,13 @@ public class HMaster extends HRegionServer implements MasterServices {
}
@Override
- public LoadBalancer getLoadBalancer() {
+ public RSGroupBasedLoadBalancer getLoadBalancer() {
return balancer;
}
@Override
public FavoredNodesManager getFavoredNodesManager() {
- return favoredNodesManager;
+ return balancer.getFavoredNodesManager();
}
private long executePeerProcedure(AbstractPeerProcedure<?> procedure) throws IOException {
@@ -3875,7 +3870,7 @@ public class HMaster extends HRegionServer implements MasterServices {
}
@Override
- public RSGroupInfoManager getRSRSGroupInfoManager() {
+ public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
index 0fc544a..d5ca1f7 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
@@ -65,6 +65,15 @@ public interface LoadBalancer extends Configurable, Stoppable, ConfigurationObse
ServerName BOGUS_SERVER_NAME = ServerName.valueOf("localhost,1,1");
/**
+ * Config for pluggable load balancers.
+ * @deprecated since 3.0.0, will be removed in 4.0.0. In the new implementation, as the base load
+ * balancer will always be the rs group based one, you should just use
+ * {@link org.apache.hadoop.hbase.HConstants#HBASE_MASTER_LOADBALANCER_CLASS} to
+ * config the per group load balancer.
+ */
+ @Deprecated
+ String HBASE_RSGROUP_LOADBALANCER_CLASS = "hbase.rsgroup.grouploadbalancer.class";
+ /**
* Set the current cluster status. This allows a LoadBalancer to map host name to a server
*/
void setClusterMetrics(ClusterMetrics st);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
index f5a89d7..af408d6 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
@@ -75,6 +75,7 @@ import org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil;
import org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.NonceProcedureRunnable;
import org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure;
import org.apache.hadoop.hbase.mob.MobUtils;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.procedure.MasterProcedureManager;
import org.apache.hadoop.hbase.procedure2.LockType;
import org.apache.hadoop.hbase.procedure2.LockedResource;
@@ -220,10 +221,8 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsNormaliz
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsNormalizerEnabledResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsProcedureDoneRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsProcedureDoneResponse;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .IsSnapshotCleanupEnabledRequest;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .IsSnapshotCleanupEnabledResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotCleanupEnabledRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotCleanupEnabledResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotDoneRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSnapshotDoneResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.IsSplitOrMergeEnabledRequest;
@@ -276,10 +275,8 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetNormali
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetNormalizerRunningResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetQuotaRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetQuotaResponse;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .SetSnapshotCleanupRequest;
-import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos
- .SetSnapshotCleanupResponse;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSnapshotCleanupRequest;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSnapshotCleanupResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSplitOrMergeEnabledRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetSplitOrMergeEnabledResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetTableStateInMetaRequest;
@@ -2382,12 +2379,18 @@ public class MasterRpcServices extends RSRpcServices
LOG.debug("Some dead server is still under processing, won't clear the dead server list");
response.addAllServerName(request.getServerNameList());
} else {
+ DeadServer deadServer = master.getServerManager().getDeadServers();
+ Set<Address> clearedServers = new HashSet<>();
for (HBaseProtos.ServerName pbServer : request.getServerNameList()) {
- if (!master.getServerManager().getDeadServers()
- .removeDeadServer(ProtobufUtil.toServerName(pbServer))) {
+ ServerName server = ProtobufUtil.toServerName(pbServer);
+ if (!deadServer.removeDeadServer(server)) {
response.addServerName(pbServer);
+ } else {
+ clearedServers.add(server.getAddress());
}
}
+ master.getRSGroupInfoManager().removeServers(clearedServers);
+ LOG.info("Remove decommissioned servers {} from RSGroup done", clearedServers);
}
if (master.cpHost != null) {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
index ead40b9..ae55352 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
@@ -539,5 +539,5 @@ public interface MasterServices extends Server {
/**
* @return the {@link RSGroupInfoManager}
*/
- RSGroupInfoManager getRSRSGroupInfoManager();
+ RSGroupInfoManager getRSGroupInfoManager();
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
index debdf68..0f383bd 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
@@ -68,6 +68,7 @@ import org.apache.hadoop.hbase.procedure2.ProcedureExecutor;
import org.apache.hadoop.hbase.procedure2.ProcedureInMemoryChore;
import org.apache.hadoop.hbase.procedure2.util.StringUtils;
import org.apache.hadoop.hbase.regionserver.SequenceId;
+import org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.hadoop.hbase.util.HasThread;
@@ -325,6 +326,11 @@ public class AssignmentManager {
return master.getLoadBalancer();
}
+ private FavoredNodesPromoter getFavoredNodePromoter() {
+ return (FavoredNodesPromoter) ((RSGroupBasedLoadBalancer) master.getLoadBalancer())
+ .getInternalBalancer();
+ }
+
private MasterProcedureEnv getProcedureEnvironment() {
return master.getMasterProcedureExecutor().getEnvironment();
}
@@ -368,7 +374,7 @@ public class AssignmentManager {
public List<ServerName> getFavoredNodes(final RegionInfo regionInfo) {
return this.shouldAssignRegionsWithFavoredNodes
- ? ((FavoredStochasticBalancer) getBalancer()).getFavoredNodes(regionInfo)
+ ? getFavoredNodePromoter().getFavoredNodes(regionInfo)
: ServerName.EMPTY_SERVER_LIST;
}
@@ -1774,8 +1780,8 @@ public class AssignmentManager {
regionStateStore.splitRegion(parent, daughterA, daughterB, serverName);
if (shouldAssignFavoredNodes(parent)) {
List<ServerName> onlineServers = this.master.getServerManager().getOnlineServersList();
- ((FavoredNodesPromoter)getBalancer()).
- generateFavoredNodesForDaughter(onlineServers, parent, daughterA, daughterB);
+ getFavoredNodePromoter().generateFavoredNodesForDaughter(onlineServers, parent, daughterA,
+ daughterB);
}
}
@@ -1800,8 +1806,7 @@ public class AssignmentManager {
}
regionStateStore.mergeRegions(child, mergeParents, serverName);
if (shouldAssignFavoredNodes(child)) {
- ((FavoredNodesPromoter)getBalancer()).
- generateFavoredNodesForMergedRegion(child, mergeParents);
+ getFavoredNodePromoter().generateFavoredNodesForMergedRegion(child, mergeParents);
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredStochasticBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredStochasticBalancer.java
index 1daa9e7..442dc44 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredStochasticBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredStochasticBalancer.java
@@ -473,6 +473,7 @@ public class FavoredStochasticBalancer extends StochasticLoadBalancer implements
}
}
+ @Override
public synchronized List<ServerName> getFavoredNodes(RegionInfo regionInfo) {
return this.fnm.getFavoredNodes(regionInfo);
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
index 9ce020b..bfda12e 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
@@ -17,11 +17,11 @@
*/
package org.apache.hadoop.hbase.master.balancer;
-import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.master.LoadBalancer;
-import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.hbase.util.ReflectionUtils;
+import org.apache.yetus.audience.InterfaceAudience;
/**
* The class that creates a load balancer from a conf.
@@ -30,8 +30,7 @@ import org.apache.hadoop.util.ReflectionUtils;
public class LoadBalancerFactory {
/**
- * The default {@link LoadBalancer} class.
- *
+ * The default {@link LoadBalancer} class.
* @return The Class for the default {@link LoadBalancer}.
*/
public static Class<? extends LoadBalancer> getDefaultLoadBalancerClass() {
@@ -40,16 +39,15 @@ public class LoadBalancerFactory {
/**
* Create a loadbalancer from the given conf.
- * @param conf
* @return A {@link LoadBalancer}
*/
public static LoadBalancer getLoadBalancer(Configuration conf) {
-
// Create the balancer
Class<? extends LoadBalancer> balancerKlass =
- conf.getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, getDefaultLoadBalancerClass(),
- LoadBalancer.class);
- return ReflectionUtils.newInstance(balancerKlass, conf);
-
+ conf.getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, getDefaultLoadBalancerClass(),
+ LoadBalancer.class);
+ LoadBalancer balancer = ReflectionUtils.newInstance(balancerKlass);
+ balancer.setConf(conf);
+ return balancer;
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AbstractStateMachineNamespaceProcedure.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AbstractStateMachineNamespaceProcedure.java
index e751034..5c35544 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AbstractStateMachineNamespaceProcedure.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AbstractStateMachineNamespaceProcedure.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.hbase.master.procedure;
import java.io.IOException;
+import java.util.function.Supplier;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.master.MasterFileSystem;
@@ -28,6 +29,7 @@ import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hbase.procedure2.StateMachineProcedure;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.util.FSUtils;
/**
@@ -122,4 +124,13 @@ public abstract class AbstractStateMachineNamespaceProcedure<TState>
protected void releaseSyncLatch() {
ProcedurePrepareLatch.releaseLatch(syncLatch, this);
}
+
+ protected final void checkNamespaceRSGroup(MasterProcedureEnv env, NamespaceDescriptor nd)
+ throws IOException {
+ Supplier<String> forWhom = () -> "namespace " + nd.getName();
+ RSGroupInfo rsGroupInfo = MasterProcedureUtil.checkGroupExists(
+ env.getMasterServices().getRSGroupInfoManager()::getRSGroup,
+ MasterProcedureUtil.getNamespaceGroup(nd), forWhom);
+ MasterProcedureUtil.checkGroupNotEmpty(rsGroupInfo, forWhom);
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java
index 28f7585..bc60360 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java
@@ -188,6 +188,7 @@ public class CreateNamespaceProcedure
return false;
}
getTableNamespaceManager(env).validateTableAndRegionCount(nsDescriptor);
+ checkNamespaceRSGroup(env, nsDescriptor);
return true;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index 34fde27..5cdfede 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -21,11 +21,12 @@ package org.apache.hadoop.hbase.master.procedure;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
-
+import java.util.function.Supplier;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.TableExistsException;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
@@ -35,6 +36,7 @@ import org.apache.hadoop.hbase.client.TableState;
import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
import org.apache.hadoop.hbase.master.MasterFileSystem;
import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.util.FSTableDescriptors;
import org.apache.hadoop.hbase.util.FSUtils;
import org.apache.hadoop.hbase.util.ModifyRegionUtils;
@@ -42,8 +44,10 @@ import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
+
import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos;
@@ -254,10 +258,27 @@ public class CreateTableProcedure
// check that we have at least 1 CF
if (tableDescriptor.getColumnFamilyCount() == 0) {
- setFailure("master-create-table", new DoNotRetryIOException("Table " +
- getTableName().toString() + " should have at least one column family."));
+ setFailure("master-create-table", new DoNotRetryIOException(
+ "Table " + getTableName().toString() + " should have at least one column family."));
return false;
}
+ if (!tableName.isSystemTable()) {
+ // do not check rs group for system tables as we may block the bootstrap.
+ Supplier<String> forWhom = () -> "table " + tableName;
+ RSGroupInfo rsGroupInfo = MasterProcedureUtil.checkGroupExists(
+ env.getMasterServices().getRSGroupInfoManager()::getRSGroup,
+ tableDescriptor.getRegionServerGroup(), forWhom);
+ if (rsGroupInfo == null) {
+ // we do not set rs group info on table, check if we have one on namespace
+ String namespace = tableName.getNamespaceAsString();
+ NamespaceDescriptor nd = env.getMasterServices().getClusterSchema().getNamespace(namespace);
+ forWhom = () -> "table " + tableName + "(inherit from namespace)";
+ rsGroupInfo = MasterProcedureUtil.checkGroupExists(
+ env.getMasterServices().getRSGroupInfoManager()::getRSGroup,
+ MasterProcedureUtil.getNamespaceGroup(nd), forWhom);
+ }
+ MasterProcedureUtil.checkGroupNotEmpty(rsGroupInfo, forWhom);
+ }
return true;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
index 1e488b6..df8fa2b 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureUtil.java
@@ -18,14 +18,18 @@
package org.apache.hadoop.hbase.master.procedure;
import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Supplier;
import java.util.regex.Pattern;
-
import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.procedure2.Procedure;
import org.apache.hadoop.hbase.procedure2.ProcedureException;
import org.apache.hadoop.hbase.procedure2.ProcedureExecutor;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.util.NonceKey;
import org.apache.hadoop.security.UserGroupInformation;
@@ -186,10 +190,52 @@ public final class MasterProcedureUtil {
* keep trying. The default proc.getException().unwrapRemoteException
* doesn't have access to DNRIOE from the procedure2 module.
*/
- public static IOException unwrapRemoteIOException(Procedure proc) {
+ public static IOException unwrapRemoteIOException(Procedure<?> proc) {
Exception e = proc.getException().unwrapRemoteException();
// Do not retry ProcedureExceptions!
return (e instanceof ProcedureException)? new DoNotRetryIOException(e):
proc.getException().unwrapRemoteIOException();
}
+
+ /**
+ * Do not allow creating new tables/namespaces which has an empty rs group, expect the default rs
+ * group. Notice that we do not check for online servers, as this is not stable because region
+ * servers can die at any time.
+ */
+ public static void checkGroupNotEmpty(RSGroupInfo rsGroupInfo, Supplier<String> forWhom)
+ throws ConstraintException {
+ if (rsGroupInfo == null || rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ // we do not have a rs group config or we explicitly set the rs group to default, then no need
+ // to check.
+ return;
+ }
+ if (rsGroupInfo.getServers().isEmpty()) {
+ throw new ConstraintException(
+ "No servers in the rsgroup " + rsGroupInfo.getName() + " for " + forWhom.get());
+ }
+ }
+
+ @FunctionalInterface
+ public interface RSGroupGetter {
+ RSGroupInfo get(String groupName) throws IOException;
+ }
+
+ public static RSGroupInfo checkGroupExists(RSGroupGetter getter, Optional<String> optGroupName,
+ Supplier<String> forWhom) throws IOException {
+ if (optGroupName.isPresent()) {
+ String groupName = optGroupName.get();
+ RSGroupInfo group = getter.get(groupName);
+ if (group == null) {
+ throw new ConstraintException(
+ "Region server group " + groupName + " for " + forWhom.get() + " does not exit");
+ }
+ return group;
+ }
+ return null;
+ }
+
+ public static Optional<String> getNamespaceGroup(NamespaceDescriptor namespaceDesc) {
+ return Optional
+ .ofNullable(namespaceDesc.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP));
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java
index e3327e2..c52b93d 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java
@@ -18,10 +18,11 @@
package org.apache.hadoop.hbase.master.procedure;
import java.io.IOException;
+import java.util.Objects;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.NamespaceNotFoundException;
-import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -171,24 +172,22 @@ public class ModifyNamespaceProcedure
/**
* Action before any real action of adding namespace.
- * @param env MasterProcedureEnv
- * @throws IOException
*/
private boolean prepareModify(final MasterProcedureEnv env) throws IOException {
- if (getTableNamespaceManager(env).doesNamespaceExist(newNsDescriptor.getName()) == false) {
+ if (!getTableNamespaceManager(env).doesNamespaceExist(newNsDescriptor.getName())) {
setFailure("master-modify-namespace",
new NamespaceNotFoundException(newNsDescriptor.getName()));
return false;
}
- try {
- getTableNamespaceManager(env).validateTableAndRegionCount(newNsDescriptor);
- } catch (ConstraintException e) {
- setFailure("master-modify-namespace", e);
- return false;
- }
+ getTableNamespaceManager(env).validateTableAndRegionCount(newNsDescriptor);
// This is used for rollback
oldNsDescriptor = getTableNamespaceManager(env).get(newNsDescriptor.getName());
+ if (!Objects.equals(
+ oldNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP),
+ newNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP))) {
+ checkNamespaceRSGroup(env, newNsDescriptor);
+ }
return true;
}
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
index 8e435dd..3f26807 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
@@ -22,7 +22,7 @@ import java.io.IOException;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-
+import java.util.function.Supplier;
import org.apache.hadoop.hbase.ConcurrentTableModificationException;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.HBaseIOException;
@@ -41,6 +41,7 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableState;
import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.util.ServerRegionReplicaUtil;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
@@ -242,8 +243,6 @@ public class ModifyTableProcedure
/**
* Check conditions before any real action of modifying a table.
- * @param env MasterProcedureEnv
- * @throws IOException
*/
private void prepareModify(final MasterProcedureEnv env) throws IOException {
// Checks whether the table exists
@@ -285,6 +284,8 @@ public class ModifyTableProcedure
}
}
+
+
// Find out whether all column families in unmodifiedTableDescriptor also exists in
// the modifiedTableDescriptor. This is to determine whether we are safe to rollback.
final Set<byte[]> oldFamilies = unmodifiedTableDescriptor.getColumnFamilyNames();
@@ -295,6 +296,14 @@ public class ModifyTableProcedure
break;
}
}
+ if (!unmodifiedTableDescriptor.getRegionServerGroup()
+ .equals(modifiedTableDescriptor.getRegionServerGroup())) {
+ Supplier<String> forWhom = () -> "table " + getTableName();
+ RSGroupInfo rsGroupInfo = MasterProcedureUtil.checkGroupExists(
+ env.getMasterServices().getRSGroupInfoManager()::getRSGroup,
+ modifiedTableDescriptor.getRegionServerGroup(), forWhom);
+ MasterProcedureUtil.checkGroupNotEmpty(rsGroupInfo, forWhom);
+ }
}
/**
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 0bde67b..353b4d2 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -20,38 +20,25 @@ package org.apache.hadoop.hbase.rsgroup;
import com.google.protobuf.Service;
import java.io.IOException;
import java.util.Collections;
-import java.util.List;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.Set;
-import java.util.function.Supplier;
-import java.util.stream.Collectors;
import org.apache.hadoop.hbase.CoprocessorEnvironment;
-import org.apache.hadoop.hbase.HConstants;
-import org.apache.hadoop.hbase.NamespaceDescriptor;
-import org.apache.hadoop.hbase.ServerName;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.RegionInfo;
-import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.coprocessor.CoreCoprocessor;
import org.apache.hadoop.hbase.coprocessor.HasMasterServices;
import org.apache.hadoop.hbase.coprocessor.MasterCoprocessor;
-import org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment;
-import org.apache.hadoop.hbase.coprocessor.MasterObserver;
-import org.apache.hadoop.hbase.coprocessor.ObserverContext;
import org.apache.hadoop.hbase.master.MasterServices;
-import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.security.UserProvider;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
-// TODO: Encapsulate MasterObserver functions into separate subclass.
+/**
+ * @deprecated Keep it here only for compatibility with old client, all the logics have been moved
+ * into core of HBase.
+ */
+@Deprecated
@CoreCoprocessor
@InterfaceAudience.Private
-public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
+public class RSGroupAdminEndpoint implements MasterCoprocessor {
// Only instance of RSGroupInfoManager. RSGroup aware load balancers ask for this instance on
// their setup.
private MasterServices master;
@@ -66,13 +53,8 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
}
master = ((HasMasterServices) env).getMasterServices();
- groupInfoManager = master.getRSRSGroupInfoManager();
+ groupInfoManager = master.getRSGroupInfoManager();
groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
- Class<?> clazz =
- master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
- if (!RSGroupableBalancer.class.isAssignableFrom(clazz)) {
- throw new IOException("Configured balancer does not support RegionServer groups.");
- }
AccessChecker accessChecker = ((HasMasterServices) env).getMasterServices().getAccessChecker();
// set the user-provider.
@@ -89,11 +71,6 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
return Collections.singleton(groupAdminService);
}
- @Override
- public Optional<MasterObserver> getMasterObserver() {
- return Optional.of(this);
- }
-
RSGroupInfoManager getGroupInfoManager() {
return groupInfoManager;
}
@@ -102,108 +79,4 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
RSGroupAdminServiceImpl getGroupAdminService() {
return groupAdminService;
}
-
- /////////////////////////////////////////////////////////////////////////////
- // MasterObserver overrides
- /////////////////////////////////////////////////////////////////////////////
-
- @Override
- public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
- List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
- Set<Address> clearedServer =
- servers.stream().filter(server -> !notClearedServers.contains(server))
- .map(ServerName::getAddress).collect(Collectors.toSet());
- if (!clearedServer.isEmpty()) {
- groupAdminServer.removeServers(clearedServer);
- }
- }
-
- private RSGroupInfo checkGroupExists(Optional<String> optGroupName, Supplier<String> forWhom)
- throws IOException {
- if (optGroupName.isPresent()) {
- String groupName = optGroupName.get();
- RSGroupInfo group = groupAdminServer.getRSGroupInfo(groupName);
- if (group == null) {
- throw new ConstraintException(
- "Region server group " + groupName + " for " + forWhom.get() + " does not exit");
- }
- return group;
- }
- return null;
- }
-
- private Optional<String> getNamespaceGroup(NamespaceDescriptor namespaceDesc) {
- return Optional
- .ofNullable(namespaceDesc.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP));
- }
-
- // Do not allow creating new tables/namespaces which has an empty rs group, expect the default rs
- // group. Notice that we do not check for online servers, as this is not stable because region
- // servers can die at any time.
- private void checkGroupNotEmpty(RSGroupInfo rsGroupInfo, Supplier<String> forWhom)
- throws ConstraintException {
- if (rsGroupInfo == null || rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
- // we do not have a rs group config or we explicitly set the rs group to default, then no need
- // to check.
- return;
- }
- if (rsGroupInfo.getServers().isEmpty()) {
- throw new ConstraintException(
- "No servers in the rsgroup " + rsGroupInfo.getName() + " for " + forWhom.get());
- }
- }
-
- @Override
- public void preCreateTableAction(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableDescriptor desc, RegionInfo[] regions) throws IOException {
- if (desc.getTableName().isSystemTable()) {
- // do not check for system tables as we may block the bootstrap.
- return;
- }
- Supplier<String> forWhom = () -> "table " + desc.getTableName();
- RSGroupInfo rsGroupInfo = checkGroupExists(desc.getRegionServerGroup(), forWhom);
- if (rsGroupInfo == null) {
- // we do not set rs group info on table, check if we have one on namespace
- String namespace = desc.getTableName().getNamespaceAsString();
- NamespaceDescriptor nd = master.getClusterSchema().getNamespace(namespace);
- forWhom = () -> "table " + desc.getTableName() + "(inherit from namespace)";
- rsGroupInfo = checkGroupExists(getNamespaceGroup(nd), forWhom);
- }
- checkGroupNotEmpty(rsGroupInfo, forWhom);
- }
-
- @Override
- public TableDescriptor preModifyTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableName tableName, TableDescriptor currentDescriptor, TableDescriptor newDescriptor)
- throws IOException {
- if (!currentDescriptor.getRegionServerGroup().equals(newDescriptor.getRegionServerGroup())) {
- Supplier<String> forWhom = () -> "table " + newDescriptor.getTableName();
- RSGroupInfo rsGroupInfo = checkGroupExists(newDescriptor.getRegionServerGroup(), forWhom);
- checkGroupNotEmpty(rsGroupInfo, forWhom);
- }
- return MasterObserver.super.preModifyTable(ctx, tableName, currentDescriptor, newDescriptor);
- }
-
- private void checkNamespaceGroup(NamespaceDescriptor nd) throws IOException {
- Supplier<String> forWhom = () -> "namespace " + nd.getName();
- RSGroupInfo rsGroupInfo = checkGroupExists(getNamespaceGroup(nd), forWhom);
- checkGroupNotEmpty(rsGroupInfo, forWhom);
- }
-
- @Override
- public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor ns) throws IOException {
- checkNamespaceGroup(ns);
- }
-
- @Override
- public void preModifyNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor currentNsDescriptor, NamespaceDescriptor newNsDescriptor)
- throws IOException {
- if (!Objects.equals(
- currentNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP),
- newNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP))) {
- checkNamespaceGroup(newNsDescriptor);
- }
- }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index 0f943d0..fb3db84 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -35,13 +35,15 @@ import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.constraint.ConstraintException;
+import org.apache.hadoop.hbase.favored.FavoredNodesManager;
+import org.apache.hadoop.hbase.favored.FavoredNodesPromoter;
import org.apache.hadoop.hbase.master.LoadBalancer;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.master.RegionPlan;
-import org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer;
+import org.apache.hadoop.hbase.master.balancer.LoadBalancerFactory;
import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.util.Pair;
-import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.hadoop.hbase.util.ReflectionUtils;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -67,12 +69,13 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
* providing appropriate assignments for user tables.
*/
@InterfaceAudience.Private
-public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
+public class RSGroupBasedLoadBalancer implements LoadBalancer {
private static final Logger LOG = LoggerFactory.getLogger(RSGroupBasedLoadBalancer.class);
private Configuration config;
private ClusterMetrics clusterStatus;
private MasterServices masterServices;
+ private FavoredNodesManager favoredNodesManager;
private volatile RSGroupInfoManager rsGroupInfoManager;
private LoadBalancer internalBalancer;
@@ -330,36 +333,42 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
@Override
public void initialize() throws IOException {
- try {
+ if (rsGroupInfoManager == null) {
+ rsGroupInfoManager = masterServices.getRSGroupInfoManager();
if (rsGroupInfoManager == null) {
- List<RSGroupAdminEndpoint> cps =
- masterServices.getMasterCoprocessorHost().findCoprocessors(RSGroupAdminEndpoint.class);
- if (cps.size() != 1) {
- String msg = "Expected one implementation of GroupAdminEndpoint but found " + cps.size();
- LOG.error(msg);
- throw new HBaseIOException(msg);
- }
- rsGroupInfoManager = cps.get(0).getGroupInfoManager();
- if(rsGroupInfoManager == null){
- String msg = "RSGroupInfoManager hasn't been initialized";
- LOG.error(msg);
- throw new HBaseIOException(msg);
- }
- rsGroupInfoManager.start();
+ String msg = "RSGroupInfoManager hasn't been initialized";
+ LOG.error(msg);
+ throw new HBaseIOException(msg);
}
- } catch (IOException e) {
- throw new HBaseIOException("Failed to initialize GroupInfoManagerImpl", e);
+ rsGroupInfoManager.start();
}
// Create the balancer
- Class<? extends LoadBalancer> balancerKlass = config.getClass(HBASE_RSGROUP_LOADBALANCER_CLASS,
- StochasticLoadBalancer.class, LoadBalancer.class);
- internalBalancer = ReflectionUtils.newInstance(balancerKlass, config);
+ Class<? extends LoadBalancer> balancerClass;
+ String balancerClassName = config.get(HBASE_RSGROUP_LOADBALANCER_CLASS);
+ if (balancerClassName == null) {
+ balancerClass = config.getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
+ LoadBalancerFactory.getDefaultLoadBalancerClass(), LoadBalancer.class);
+ } else {
+ try {
+ balancerClass = Class.forName(balancerClassName).asSubclass(LoadBalancer.class);
+ } catch (ClassNotFoundException e) {
+ throw new IOException(e);
+ }
+ }
+ // avoid infinite nesting
+ if (getClass().isAssignableFrom(balancerClass)) {
+ balancerClass = LoadBalancerFactory.getDefaultLoadBalancerClass();
+ }
+ internalBalancer = ReflectionUtils.newInstance(balancerClass);
+ if (internalBalancer instanceof FavoredNodesPromoter) {
+ favoredNodesManager = new FavoredNodesManager(masterServices);
+ }
+ internalBalancer.setConf(config);
internalBalancer.setMasterServices(masterServices);
if(clusterStatus != null) {
internalBalancer.setClusterMetrics(clusterStatus);
}
- internalBalancer.setConf(config);
internalBalancer.initialize();
}
@@ -402,6 +411,14 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
this.rsGroupInfoManager = rsGroupInfoManager;
}
+ public LoadBalancer getInternalBalancer() {
+ return internalBalancer;
+ }
+
+ public FavoredNodesManager getFavoredNodesManager() {
+ return favoredNodesManager;
+ }
+
@Override
public void postMasterStartupInitialize() {
this.internalBalancer.postMasterStartupInitialize();
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
deleted file mode 100644
index d091b3c..0000000
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hbase.rsgroup;
-
-import org.apache.hadoop.hbase.master.LoadBalancer;
-import org.apache.yetus.audience.InterfaceAudience;
-
-/**
- * Marker Interface. RSGroups feature will check for a LoadBalancer
- * marked with this Interface before it runs.
- */
-@InterfaceAudience.Private
-public interface RSGroupableBalancer extends LoadBalancer {
- /** Config for pluggable load balancers */
- String HBASE_RSGROUP_LOADBALANCER_CLASS = "hbase.rsgroup.grouploadbalancer.class";
-}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
index 1fe43f1..df39862 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
@@ -115,7 +115,7 @@ public class TestNamespace {
//verify existence of system tables
Set<TableName> systemTables = Sets.newHashSet(
- TableName.META_TABLE_NAME);
+ TableName.META_TABLE_NAME, TableName.valueOf("hbase:rsgroup"));
List<TableDescriptor> descs = admin.listTableDescriptorsByNamespace(
Bytes.toBytes(NamespaceDescriptor.SYSTEM_NAMESPACE.getName()));
assertEquals(systemTables.size(), descs.size());
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
index a576adc..fc2e9f7 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
@@ -504,14 +504,14 @@ public abstract class AbstractTestDLS {
for (String oregion : regions)
LOG.debug("Region still online: " + oregion);
}
- assertEquals(1 + existingRegions, regions.size());
+ assertEquals(2 + existingRegions, regions.size());
LOG.debug("Enabling table\n");
TEST_UTIL.getAdmin().enableTable(tableName);
LOG.debug("Waiting for no more RIT\n");
blockUntilNoRIT();
LOG.debug("Verifying there are " + numRegions + " assigned on cluster\n");
regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
- assertEquals(numRegions + 1 + existingRegions, regions.size());
+ assertEquals(numRegions + 2 + existingRegions, regions.size());
return table;
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
index 26772ae..71f8a91 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
@@ -493,7 +493,7 @@ public class MockNoopMasterServices implements MasterServices {
}
@Override
- public RSGroupInfoManager getRSRSGroupInfoManager() {
+ public RSGroupInfoManager getRSGroupInfoManager() {
return null;
}
}
\ No newline at end of file
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
index 5a38234..a8d80b6 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
@@ -43,6 +43,8 @@ public class TestClusterRestart extends AbstractTestRestartCluster {
private static final Logger LOG = LoggerFactory.getLogger(TestClusterRestart.class);
+ private static final int NUM_REGIONS = 4;
+
@Override
protected boolean splitWALCoordinatedByZk() {
return true;
@@ -61,7 +63,7 @@ public class TestClusterRestart extends AbstractTestRestartCluster {
}
List<RegionInfo> allRegions = MetaTableAccessor.getAllRegions(UTIL.getConnection(), false);
- assertEquals(3, allRegions.size());
+ assertEquals(NUM_REGIONS, allRegions.size());
LOG.info("\n\nShutting down cluster");
UTIL.shutdownMiniHBaseCluster();
@@ -76,7 +78,7 @@ public class TestClusterRestart extends AbstractTestRestartCluster {
// Otherwise we're reusing an Connection that has gone stale because
// the shutdown of the cluster also called shut of the connection.
allRegions = MetaTableAccessor.getAllRegions(UTIL.getConnection(), false);
- assertEquals(3, allRegions.size());
+ assertEquals(NUM_REGIONS, allRegions.size());
LOG.info("\n\nWaiting for tables to be available");
for (TableName TABLE : TABLES) {
try {
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
index 75d9ee1..61720df 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
@@ -25,6 +25,7 @@ import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.MiniHBaseCluster;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.StartMiniClusterOption;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.test.MetricsAssertHelper;
import org.apache.hadoop.hbase.testclassification.MasterTests;
import org.apache.hadoop.hbase.testclassification.MediumTests;
@@ -90,6 +91,7 @@ public class TestMasterMetrics {
cluster = TEST_UTIL.getHBaseCluster();
LOG.info("Waiting for active/ready master");
cluster.waitForActiveAndReadyMaster();
+ TEST_UTIL.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
master = cluster.getMaster();
}
@@ -131,7 +133,7 @@ public class TestMasterMetrics {
MetricsMasterSource masterSource = master.getMasterMetrics().getMetricsSource();
boolean tablesOnMaster = LoadBalancer.isTablesOnMaster(TEST_UTIL.getConfiguration());
metricsHelper.assertGauge("numRegionServers", 1 + (tablesOnMaster ? 1 : 0), masterSource);
- metricsHelper.assertGauge("averageLoad", 1, masterSource);
+ metricsHelper.assertGauge("averageLoad", 2, masterSource);
metricsHelper.assertGauge("numDeadRegionServers", 0, masterSource);
metricsHelper.assertGauge("masterStartTime", master.getMasterStartTime(), masterSource);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
index fca2866..2d0af3e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
@@ -92,7 +92,7 @@ public class TestMasterRestartAfterDisablingTable {
NavigableSet<String> regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
assertEquals("The number of regions for the table tableRestart should be 0 and only" +
- "the catalog table should be present.", 1, regions.size());
+ "the catalog table should be present.", 2, regions.size());
List<MasterThread> masterThreads = cluster.getMasterThreads();
MasterThread activeMaster = null;
@@ -120,7 +120,7 @@ public class TestMasterRestartAfterDisablingTable {
log("Verifying there are " + numRegions + " assigned on cluster\n");
regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
assertEquals("The assigned regions were not onlined after master" +
- " switch except for the catalog table.", 5, regions.size());
+ " switch except for the catalog table.", 6, regions.size());
assertTrue("The table should be in enabled state", cluster.getMaster().getTableStateManager()
.isTableState(TableName.valueOf(name.getMethodName()), TableState.State.ENABLED));
ht.close();
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
index 0aba487..2dc1b6c 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
@@ -121,7 +121,7 @@ public class TestRollingRestart {
log("Region still online: " + oregion);
}
}
- assertEquals(1, regions.size());
+ assertEquals(2, regions.size());
log("Enabling table\n");
TEST_UTIL.getAdmin().enableTable(tableName);
log("Waiting for no more RIT\n");
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticBalancerPickers.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticBalancerPickers.java
index 0aa2322..7677eb6 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticBalancerPickers.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestFavoredStochasticBalancerPickers.java
@@ -186,7 +186,7 @@ public class TestFavoredStochasticBalancerPickers extends BalancerTestBase {
regionFinder.setServices(TEST_UTIL.getMiniHBaseCluster().getMaster());
Cluster cluster = new Cluster(serverAssignments, null, regionFinder, new RackManager(conf));
LoadOnlyFavoredStochasticBalancer balancer = (LoadOnlyFavoredStochasticBalancer) TEST_UTIL
- .getMiniHBaseCluster().getMaster().getLoadBalancer();
+ .getMiniHBaseCluster().getMaster().getLoadBalancer().getInternalBalancer();
cluster.sortServersByRegionCount();
Integer[] servers = cluster.serverIndicesSortedByRegionCount;
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
index 7ab5dcc..2503825 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
@@ -59,7 +59,6 @@ public class TestRegionOpen {
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestRegionOpen.class);
- @SuppressWarnings("unused")
private static final Logger LOG = LoggerFactory.getLogger(TestRegionOpen.class);
private static final int NB_SERVERS = 1;
@@ -87,6 +86,7 @@ public class TestRegionOpen {
final TableName tableName = TableName.valueOf(TestRegionOpen.class.getSimpleName());
ThreadPoolExecutor exec = getRS().getExecutorService()
.getExecutorThreadPool(ExecutorType.RS_OPEN_PRIORITY_REGION);
+ HTU.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
long completed = exec.getCompletedTaskCount();
HTableDescriptor htd = new HTableDescriptor(tableName);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
index 70df61e..dc0a918 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
@@ -146,7 +146,7 @@ public class TestRegionReplicasWithRestartScenarios {
checkDuplicates(onlineRegions3);
assertFalse(res);
int totalRegions = onlineRegions.size() + onlineRegions2.size() + onlineRegions3.size();
- assertEquals(61, totalRegions);
+ assertEquals(62, totalRegions);
}
private boolean checkDuplicates(Collection<HRegion> onlineRegions3) throws Exception {
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
index bc44daa..952bcbc 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
@@ -225,7 +225,7 @@ public class TestRegionServerMetrics {
@Test
public void testRegionCount() throws Exception {
- metricsHelper.assertGauge("regionCount", TABLES_ON_MASTER ? 1 : 2, serverSource);
+ metricsHelper.assertGauge("regionCount", TABLES_ON_MASTER ? 1 : 3, serverSource);
}
@Test
@@ -335,7 +335,7 @@ public class TestRegionServerMetrics {
TEST_UTIL.getAdmin().flush(tableName);
metricsRegionServer.getRegionServerWrapper().forceRecompute();
- assertGauge("storeCount", TABLES_ON_MASTER ? 1 : 5);
+ assertGauge("storeCount", TABLES_ON_MASTER ? 1 : 6);
assertGauge("storeFileCount", 1);
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
index d3577f2..7ac1a49 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
@@ -163,8 +163,7 @@ public class TestRSGroupsOfflineMode {
});
// Get groupInfoManager from the new active master.
- RSGroupInfoManager groupMgr = ((MiniHBaseCluster) cluster).getMaster()
- .getMasterCoprocessorHost().findCoprocessor(RSGroupAdminEndpoint.class).getGroupInfoManager();
+ RSGroupInfoManager groupMgr = ((MiniHBaseCluster) cluster).getMaster().getRSGroupInfoManager();
// Make sure balancer is in offline mode, since this is what we're testing.
assertFalse(groupMgr.isOnline());
// Kill final regionserver to see the failover happens for all tables except GROUP table since
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
index 14ea0b3..e03a823 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
@@ -102,6 +102,7 @@ public class TestTablePermissions {
// Wait for the ACL table to become available
UTIL.waitTableEnabled(PermissionStorage.ACL_TABLE_NAME);
+ UTIL.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
ZKW = new ZKWatcher(UTIL.getConfiguration(),
"TestTablePermissions", ABORTABLE);
@@ -222,7 +223,7 @@ public class TestTablePermissions {
// check full load
Map<byte[], ListMultimap<String, UserPermission>> allPerms = PermissionStorage.loadAll(conf);
assertEquals("Full permission map should have entries for both test tables",
- 2, allPerms.size());
+ 3, allPerms.size());
userPerms = allPerms.get(TEST_TABLE.getName()).get("hubert");
assertNotNull(userPerms);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
index 316a321..0ac07d8 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
@@ -24,6 +24,7 @@ import java.util.stream.Stream;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerStorage;
import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
@@ -52,6 +53,7 @@ public class TestHBaseFsckReplication {
public static void setUp() throws Exception {
UTIL.getConfiguration().setBoolean("hbase.write.hbck1.lock.file", false);
UTIL.startMiniCluster(1);
+ UTIL.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
}
@AfterClass
[hbase] 13/14: HBASE-23050 Use RSGroupInfoManager to get rsgroups
in master UI's rsgroup part (#776)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit c1480c0724e38c6ec1d298ac883d55d14c7bd77c
Author: Guangxu Cheng <gu...@gmail.com>
AuthorDate: Thu Oct 31 03:05:52 2019 +0800
HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
Signed-off-by: Duo Zhang <zh...@apache.org>
---
.../jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon | 3 ++-
.../jamon/org/apache/hadoop/hbase/tmpl/master/RSGroupListTmpl.jamon | 2 +-
.../main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
index 8f90736..4e9b2ad 100644
--- a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
+++ b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
@@ -52,6 +52,7 @@ org.apache.hadoop.hbase.master.RegionState;
org.apache.hadoop.hbase.master.ServerManager;
org.apache.hadoop.hbase.protobuf.ProtobufUtil;
org.apache.hadoop.hbase.quotas.QuotaUtil;
+org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
org.apache.hadoop.hbase.security.access.PermissionStorage;
org.apache.hadoop.hbase.security.visibility.VisibilityConstants;
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
@@ -224,7 +225,7 @@ AssignmentManager assignmentManager = master.getAssignmentManager();
<& AssignmentManagerStatusTmpl; assignmentManager=master.getAssignmentManager()&>
</%if>
<%if !master.isInMaintenanceMode() %>
- <%if master.getMasterCoprocessorHost().findCoprocessor("RSGroupAdminEndpoint") != null &&
+ <%if master.getConfiguration().getBoolean(RSGroupInfoManager.RS_GROUP_ENABLED, false) &&
serverManager.getOnlineServersList().size() > 0 %>
<section>
<h2><a name="rsgroup">RSGroup</a></h2>
diff --git a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RSGroupListTmpl.jamon b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RSGroupListTmpl.jamon
index 61ddf09..15d0164 100644
--- a/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RSGroupListTmpl.jamon
+++ b/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/RSGroupListTmpl.jamon
@@ -40,7 +40,7 @@ ServerManager serverManager;
org.apache.hadoop.util.StringUtils.TraditionalBinaryPrefix;
</%import>
<%java>
-List<RSGroupInfo> groups = RSGroupTableAccessor.getAllRSGroupInfo(master.getConnection());
+List<RSGroupInfo> groups = master.getRSGroupInfoManager().listRSGroups();
</%java>
<%if (groups != null && groups.size() > 0)%>
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 3f73f78..0eb15e9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -32,7 +32,7 @@ import org.apache.yetus.audience.InterfaceAudience;
@InterfaceAudience.Private
public interface RSGroupInfoManager {
- static final String RS_GROUP_ENABLED = "hbase.balancer.rsgroup.enabled";
+ public static final String RS_GROUP_ENABLED = "hbase.balancer.rsgroup.enabled";
void start();
[hbase] 05/14: HBASE-22809 Allow creating table in group when rs
group contains no live servers (#464)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 83fa1e4bd7ec2052b9aae50a870b78e1750d9a8a
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Fri Aug 9 09:25:00 2019 +0800
HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 119 ++++++++++++---------
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java | 4 +-
.../hadoop/hbase/rsgroup/TestRSGroupsBasics.java | 42 --------
3 files changed, 71 insertions(+), 94 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 3c4530f..a2a5623 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -21,11 +21,12 @@ import com.google.protobuf.Service;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
+import java.util.Objects;
import java.util.Optional;
import java.util.Set;
+import java.util.function.Supplier;
import java.util.stream.Collectors;
import org.apache.hadoop.hbase.CoprocessorEnvironment;
-import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.ServerName;
@@ -68,7 +69,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
groupInfoManager = RSGroupInfoManagerImpl.getInstance(master);
groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
Class<?> clazz =
- master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
+ master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
if (!RSGroupableBalancer.class.isAssignableFrom(clazz)) {
throw new IOException("Configured balancer does not support RegionServer groups.");
}
@@ -108,85 +109,101 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
@Override
public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
- List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
+ List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
Set<Address> clearedServer =
- servers.stream().filter(server -> !notClearedServers.contains(server))
- .map(ServerName::getAddress).collect(Collectors.toSet());
+ servers.stream().filter(server -> !notClearedServers.contains(server))
+ .map(ServerName::getAddress).collect(Collectors.toSet());
if (!clearedServer.isEmpty()) {
groupAdminServer.removeServers(clearedServer);
}
}
- private void checkGroupExists(Optional<String> optGroupName) throws IOException {
+ private RSGroupInfo checkGroupExists(Optional<String> optGroupName, Supplier<String> forWhom)
+ throws IOException {
if (optGroupName.isPresent()) {
String groupName = optGroupName.get();
- if (groupAdminServer.getRSGroupInfo(groupName) == null) {
- throw new ConstraintException("Region server group " + groupName + " does not exit");
+ RSGroupInfo group = groupAdminServer.getRSGroupInfo(groupName);
+ if (group == null) {
+ throw new ConstraintException(
+ "Region server group " + groupName + " for " + forWhom.get() + " does not exit");
}
+ return group;
}
+ return null;
}
- private boolean rsgroupHasServersOnline(TableDescriptor desc) throws IOException {
- RSGroupInfo rsGroupInfo;
- Optional<String> optGroupName = desc.getRegionServerGroup();
- if (optGroupName.isPresent()) {
- String groupName = optGroupName.get();
- if (groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
- // do not check for default group
- return true;
- }
- rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo == null) {
- throw new ConstraintException(
- "RSGroup " + groupName + " for table " + desc.getTableName() + " does not exist");
- }
- } else {
- NamespaceDescriptor nd =
- master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString());
- String groupNameOfNs = nd.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (groupNameOfNs == null || groupNameOfNs.equals(RSGroupInfo.DEFAULT_GROUP)) {
- // do not check for default group
- return true;
- }
- rsGroupInfo = groupAdminServer.getRSGroupInfo(groupNameOfNs);
- if (rsGroupInfo == null) {
- throw new ConstraintException("RSGroup " + groupNameOfNs + " for table " +
- desc.getTableName() + "(inherit from namespace) does not exist");
- }
+ private Optional<String> getNamespaceGroup(NamespaceDescriptor namespaceDesc) {
+ return Optional
+ .ofNullable(namespaceDesc.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP));
+ }
+
+ // Do not allow creating new tables/namespaces which has an empty rs group, expect the default rs
+ // group. Notice that we do not check for online servers, as this is not stable because region
+ // servers can die at any time.
+ private void checkGroupNotEmpty(RSGroupInfo rsGroupInfo, Supplier<String> forWhom)
+ throws ConstraintException {
+ if (rsGroupInfo == null || rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ // we do not have a rs group config or we explicitly set the rs group to default, then no need
+ // to check.
+ return;
+ }
+ if (rsGroupInfo.getServers().isEmpty()) {
+ throw new ConstraintException(
+ "No servers in the rsgroup " + rsGroupInfo.getName() + " for " + forWhom.get());
}
- return master.getServerManager().createDestinationServersList().stream()
- .anyMatch(onlineServer -> rsGroupInfo.containsServer(onlineServer.getAddress()));
}
@Override
public void preCreateTableAction(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableDescriptor desc, RegionInfo[] regions) throws IOException {
- checkGroupExists(desc.getRegionServerGroup());
- if (!desc.getTableName().isSystemTable() && !rsgroupHasServersOnline(desc)) {
- throw new HBaseIOException("No online servers in the rsgroup for " + desc);
+ TableDescriptor desc, RegionInfo[] regions) throws IOException {
+ if (desc.getTableName().isSystemTable()) {
+ // do not check for system tables as we may block the bootstrap.
+ return;
+ }
+ Supplier<String> forWhom = () -> "table " + desc.getTableName();
+ RSGroupInfo rsGroupInfo = checkGroupExists(desc.getRegionServerGroup(), forWhom);
+ if (rsGroupInfo == null) {
+ // we do not set rs group info on table, check if we have one on namespace
+ String namespace = desc.getTableName().getNamespaceAsString();
+ NamespaceDescriptor nd = master.getClusterSchema().getNamespace(namespace);
+ forWhom = () -> "table " + desc.getTableName() + "(inherit from namespace)";
+ rsGroupInfo = checkGroupExists(getNamespaceGroup(nd), forWhom);
}
+ checkGroupNotEmpty(rsGroupInfo, forWhom);
}
@Override
public TableDescriptor preModifyTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableName tableName, TableDescriptor currentDescriptor, TableDescriptor newDescriptor)
- throws IOException {
- checkGroupExists(newDescriptor.getRegionServerGroup());
+ TableName tableName, TableDescriptor currentDescriptor, TableDescriptor newDescriptor)
+ throws IOException {
+ if (!currentDescriptor.getRegionServerGroup().equals(newDescriptor.getRegionServerGroup())) {
+ Supplier<String> forWhom = () -> "table " + newDescriptor.getTableName();
+ RSGroupInfo rsGroupInfo = checkGroupExists(newDescriptor.getRegionServerGroup(), forWhom);
+ checkGroupNotEmpty(rsGroupInfo, forWhom);
+ }
return MasterObserver.super.preModifyTable(ctx, tableName, currentDescriptor, newDescriptor);
}
+ private void checkNamespaceGroup(NamespaceDescriptor nd) throws IOException {
+ Supplier<String> forWhom = () -> "namespace " + nd.getName();
+ RSGroupInfo rsGroupInfo = checkGroupExists(getNamespaceGroup(nd), forWhom);
+ checkGroupNotEmpty(rsGroupInfo, forWhom);
+ }
+
@Override
public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor ns) throws IOException {
- checkGroupExists(
- Optional.ofNullable(ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP)));
+ NamespaceDescriptor ns) throws IOException {
+ checkNamespaceGroup(ns);
}
@Override
public void preModifyNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor currentNsDescriptor, NamespaceDescriptor newNsDescriptor)
- throws IOException {
- checkGroupExists(Optional
- .ofNullable(newNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP)));
+ NamespaceDescriptor currentNsDescriptor, NamespaceDescriptor newNsDescriptor)
+ throws IOException {
+ if (!Objects.equals(
+ currentNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP),
+ newNsDescriptor.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP))) {
+ checkNamespaceGroup(newNsDescriptor);
+ }
}
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
index 7471458..aff591f 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
@@ -152,12 +152,14 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
String nsName = tablePrefix + "_foo";
String groupName = tablePrefix + "_foo";
LOG.info("testNamespaceConstraint");
- rsGroupAdmin.addRSGroup(groupName);
+ addGroup(groupName, 1);
assertTrue(observer.preAddRSGroupCalled);
assertTrue(observer.postAddRSGroupCalled);
admin.createNamespace(NamespaceDescriptor.create(nsName)
.addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, groupName).build());
+ RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroupInfo(groupName);
+ rsGroupAdmin.moveServers(rsGroupInfo.getServers(), RSGroupInfo.DEFAULT_GROUP);
// test removing a referenced group
try {
rsGroupAdmin.removeRSGroup(groupName);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
index e3cb54e..67ed2a9 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
@@ -20,11 +20,8 @@ package org.apache.hadoop.hbase.rsgroup;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Iterator;
import java.util.List;
import java.util.Set;
import org.apache.hadoop.hbase.HBaseClassTestRule;
@@ -137,45 +134,6 @@ public class TestRSGroupsBasics extends TestRSGroupsBase {
}
@Test
- public void testCreateWhenRsgroupNoOnlineServers() throws Exception {
- LOG.info("testCreateWhenRsgroupNoOnlineServers");
-
- // set rsgroup has no online servers and test create table
- final RSGroupInfo appInfo = addGroup("appInfo", 1);
- Iterator<Address> iterator = appInfo.getServers().iterator();
- List<ServerName> serversToDecommission = new ArrayList<>();
- ServerName targetServer = getServerName(iterator.next());
- assertTrue(master.getServerManager().getOnlineServers().containsKey(targetServer));
- serversToDecommission.add(targetServer);
- admin.decommissionRegionServers(serversToDecommission, true);
- assertEquals(1, admin.listDecommissionedRegionServers().size());
-
- final TableName tableName = TableName.valueOf(tablePrefix + "_ns", name.getMethodName());
- admin.createNamespace(NamespaceDescriptor.create(tableName.getNamespaceAsString())
- .addConfiguration(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP, appInfo.getName()).build());
- final TableDescriptor desc = TableDescriptorBuilder.newBuilder(tableName)
- .setColumnFamily(ColumnFamilyDescriptorBuilder.of("f")).build();
- try {
- admin.createTable(desc);
- fail("Shouldn't create table successfully!");
- } catch (Exception e) {
- LOG.debug("create table error", e);
- }
-
- // recommission and test create table
- admin.recommissionRegionServer(targetServer, null);
- assertEquals(0, admin.listDecommissionedRegionServers().size());
- admin.createTable(desc);
- // wait for created table to be assigned
- TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
- @Override
- public boolean evaluate() throws Exception {
- return getTableRegionMap().get(desc.getTableName()) != null;
- }
- });
- }
-
- @Test
public void testDefaultNamespaceCreateAndAssign() throws Exception {
LOG.info("testDefaultNamespaceCreateAndAssign");
String tableName = tablePrefix + "_testCreateAndAssign";
[hbase] 09/14: HBASE-22987 Calculate the region servers in default
group in foreground (#599)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit cf5295abc9729e0726b41b5e7c2850e3f3ae7071
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Wed Sep 11 22:10:52 2019 +0800
HBASE-22987 Calculate the region servers in default group in foreground (#599)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 138 +++++----------------
1 file changed, 32 insertions(+), 106 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index 6725066..7224869 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -30,6 +30,7 @@ import java.util.OptionalLong;
import java.util.Set;
import java.util.SortedSet;
import java.util.TreeSet;
+import java.util.stream.Collectors;
import org.apache.hadoop.hbase.Coprocessor;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.NamespaceDescriptor;
@@ -174,8 +175,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private final RSGroupStartupWorker rsGroupStartupWorker;
// contains list of groups that were last flushed to persistent store
private Set<String> prevRSGroups = new HashSet<>();
- private final ServerEventsListenerThread serverEventsListenerThread =
- new ServerEventsListenerThread();
private RSGroupInfoManagerImpl(MasterServices masterServices) throws IOException {
this.masterServices = masterServices;
@@ -184,11 +183,34 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
this.rsGroupStartupWorker = new RSGroupStartupWorker();
}
+ private synchronized void updateDefaultServers() {
+ LOG.info("Updating default servers.");
+ Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(holder.groupName2Group);
+ RSGroupInfo oldDefaultGroupInfo = getRSGroup(RSGroupInfo.DEFAULT_GROUP);
+ assert oldDefaultGroupInfo != null;
+ RSGroupInfo newDefaultGroupInfo =
+ new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, getDefaultServers());
+ newDefaultGroupInfo.addAllTables(oldDefaultGroupInfo.getTables());
+ newGroupMap.put(RSGroupInfo.DEFAULT_GROUP, newDefaultGroupInfo);
+ // do not need to persist, as we do not persist default group.
+ resetRSGroupMap(newGroupMap);
+ LOG.info("Updated default servers, {} servers", newDefaultGroupInfo.getServers().size());
+ }
private synchronized void init() throws IOException {
refresh(false);
- serverEventsListenerThread.start();
- masterServices.getServerManager().registerListener(serverEventsListenerThread);
+ masterServices.getServerManager().registerListener(new ServerListener() {
+
+ @Override
+ public void serverAdded(ServerName serverName) {
+ updateDefaultServers();
+ }
+
+ @Override
+ public void serverRemoved(ServerName serverName) {
+ updateDefaultServers();
+ }
+ });
migrate();
}
@@ -225,19 +247,11 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
/**
- * @param master the master to get online servers for
* @return Set of online Servers named for their hostname and port (not ServerName).
*/
- private static Set<Address> getOnlineServers(final MasterServices master) {
- Set<Address> onlineServers = new HashSet<Address>();
- if (master == null) {
- return onlineServers;
- }
-
- for (ServerName server : master.getServerManager().getOnlineServers().keySet()) {
- onlineServers.add(server.getAddress());
- }
- return onlineServers;
+ private Set<Address> getOnlineServers() {
+ return masterServices.getServerManager().getOnlineServers().keySet().stream()
+ .map(ServerName::getAddress).collect(Collectors.toSet());
}
@Override
@@ -249,8 +263,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// it. If not 'default' group, add server to 'dst' rsgroup EVEN IF IT IS NOT online (could be a
// rsgroup of dead servers that are to come back later).
Set<Address> onlineServers =
- dst.getName().equals(RSGroupInfo.DEFAULT_GROUP) ? getOnlineServers(this.masterServices)
- : null;
+ dst.getName().equals(RSGroupInfo.DEFAULT_GROUP) ? getOnlineServers() : null;
for (Address el : servers) {
src.removeServer(el);
if (onlineServers != null) {
@@ -617,25 +630,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
this.prevRSGroups.addAll(currentGroups);
}
- // Called by getDefaultServers. Presume it has lock in place.
- private List<ServerName> getOnlineRS() throws IOException {
- if (masterServices != null) {
- return masterServices.getServerManager().getOnlineServersList();
- }
- LOG.debug("Reading online RS from zookeeper");
- List<ServerName> servers = new ArrayList<>();
- try {
- for (String el : ZKUtil.listChildrenNoWatch(watcher, watcher.getZNodePaths().rsZNode)) {
- servers.add(ServerName.parseServerName(el));
- }
- } catch (KeeperException e) {
- throw new IOException("Failed to retrieve server list from zookeeper", e);
- }
- return servers;
- }
-
// Called by ServerEventsListenerThread. Presume it has lock on this manager when it runs.
- private SortedSet<Address> getDefaultServers() throws IOException {
+ private SortedSet<Address> getDefaultServers() {
// Build a list of servers in other groups than default group, from rsGroupMap
Set<Address> serversInOtherGroup = new HashSet<>();
for (RSGroupInfo group : listRSGroups() /* get from rsGroupMap */) {
@@ -646,7 +642,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Get all online servers from Zookeeper and find out servers in default group
SortedSet<Address> defaultServers = Sets.newTreeSet();
- for (ServerName serverName : getOnlineRS()) {
+ for (ServerName serverName : masterServices.getServerManager().getOnlineServers().keySet()) {
Address server = Address.fromParts(serverName.getHostname(), serverName.getPort());
if (!serversInOtherGroup.contains(server)) { // not in other groups
defaultServers.add(server);
@@ -655,76 +651,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
return defaultServers;
}
- // Called by ServerEventsListenerThread. Synchronize on this because redoing
- // the rsGroupMap then writing it out.
- private synchronized void updateDefaultServers(SortedSet<Address> servers) {
- Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
- RSGroupInfo info = rsGroupMap.get(RSGroupInfo.DEFAULT_GROUP);
- RSGroupInfo newInfo = new RSGroupInfo(info.getName(), servers);
- HashMap<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
- newGroupMap.put(newInfo.getName(), newInfo);
- resetRSGroupMap(newGroupMap);
- }
-
- /**
- * Calls {@link RSGroupInfoManagerImpl#updateDefaultServers(SortedSet)} to update list of known
- * servers. Notifications about server changes are received by registering {@link ServerListener}.
- * As a listener, we need to return immediately, so the real work of updating the servers is done
- * asynchronously in this thread.
- */
- private class ServerEventsListenerThread extends Thread implements ServerListener {
- private final Logger LOG = LoggerFactory.getLogger(ServerEventsListenerThread.class);
- private boolean changed = false;
-
- ServerEventsListenerThread() {
- setDaemon(true);
- }
-
- @Override
- public void serverAdded(ServerName serverName) {
- serverChanged();
- }
-
- @Override
- public void serverRemoved(ServerName serverName) {
- serverChanged();
- }
-
- private synchronized void serverChanged() {
- changed = true;
- this.notify();
- }
-
- @Override
- public void run() {
- setName(ServerEventsListenerThread.class.getName() + "-" + masterServices.getServerName());
- SortedSet<Address> prevDefaultServers = new TreeSet<>();
- while (isMasterRunning(masterServices)) {
- try {
- LOG.info("Updating default servers.");
- SortedSet<Address> servers = RSGroupInfoManagerImpl.this.getDefaultServers();
- if (!servers.equals(prevDefaultServers)) {
- RSGroupInfoManagerImpl.this.updateDefaultServers(servers);
- prevDefaultServers = servers;
- LOG.info("Updated with servers: " + servers.size());
- }
- try {
- synchronized (this) {
- while (!changed) {
- wait();
- }
- changed = false;
- }
- } catch (InterruptedException e) {
- LOG.warn("Interrupted", e);
- }
- } catch (IOException e) {
- LOG.warn("Failed to update default servers", e);
- }
- }
- }
- }
-
private class RSGroupStartupWorker extends Thread {
private final Logger LOG = LoggerFactory.getLogger(RSGroupStartupWorker.class);
private volatile boolean online = false;
[hbase] 06/14: HBASE-22820 Do not need to persist default rs group
now (#482)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 88e361056dc8a0959cf6957b096e3603a74ce68c
Author: linkaline <li...@gmail.com>
AuthorDate: Fri Aug 16 16:52:41 2019 +0800
HBASE-22820 Do not need to persist default rs group now (#482)
Signed-off-by: Duo Zhang <zh...@apache.org>
---
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 40 +++++++++++++---------
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 25 ++++++++++++++
2 files changed, 49 insertions(+), 16 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index 37f3ce6..eaf23f3 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -396,11 +396,13 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
// populate puts
- for (RSGroupInfo RSGroupInfo : groupMap.values()) {
- RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(RSGroupInfo);
- Put p = new Put(Bytes.toBytes(RSGroupInfo.getName()));
- p.addColumn(META_FAMILY_BYTES, META_QUALIFIER_BYTES, proto.toByteArray());
- mutations.add(p);
+ for (RSGroupInfo gi : groupMap.values()) {
+ if (!gi.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(gi);
+ Put p = new Put(Bytes.toBytes(gi.getName()));
+ p.addColumn(META_FAMILY_BYTES, META_QUALIFIER_BYTES, proto.toByteArray());
+ mutations.add(p);
+ }
}
if (mutations.size() > 0) {
@@ -449,7 +451,12 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Make changes visible after having been persisted to the source of truth
resetRSGroupMap(newGroupMap);
+ saveRSGroupMapToZK(newGroupMap);
+ updateCacheOfRSGroups(newGroupMap.keySet());
+ }
+
+ private void saveRSGroupMapToZK(Map<String, RSGroupInfo> newGroupMap) throws IOException {
try {
String groupBasePath =
ZNodePaths.joinZNode(watcher.getZNodePaths().baseZNode, RS_GROUP_ZNODE);
@@ -463,14 +470,16 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
}
- for (RSGroupInfo RSGroupInfo : newGroupMap.values()) {
- String znode = ZNodePaths.joinZNode(groupBasePath, RSGroupInfo.getName());
- RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(RSGroupInfo);
- LOG.debug("Updating znode: " + znode);
- ZKUtil.createAndFailSilent(watcher, znode);
- zkOps.add(ZKUtil.ZKUtilOp.deleteNodeFailSilent(znode));
- zkOps.add(ZKUtil.ZKUtilOp.createAndFailSilent(znode,
- ProtobufUtil.prependPBMagic(proto.toByteArray())));
+ for (RSGroupInfo gi : newGroupMap.values()) {
+ if (!gi.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ String znode = ZNodePaths.joinZNode(groupBasePath, gi.getName());
+ RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(gi);
+ LOG.debug("Updating znode: " + znode);
+ ZKUtil.createAndFailSilent(watcher, znode);
+ zkOps.add(ZKUtil.ZKUtilOp.deleteNodeFailSilent(znode));
+ zkOps.add(ZKUtil.ZKUtilOp.createAndFailSilent(znode,
+ ProtobufUtil.prependPBMagic(proto.toByteArray())));
+ }
}
LOG.debug("Writing ZK GroupInfo count: " + zkOps.size());
@@ -480,7 +489,6 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
masterServices.abort("Failed to write to rsGroupZNode", e);
throw new IOException("Failed to write to rsGroupZNode", e);
}
- updateCacheOfRSGroups(newGroupMap.keySet());
}
/**
@@ -540,12 +548,12 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Called by ServerEventsListenerThread. Synchronize on this because redoing
// the rsGroupMap then writing it out.
- private synchronized void updateDefaultServers(SortedSet<Address> servers) throws IOException {
+ private synchronized void updateDefaultServers(SortedSet<Address> servers) {
RSGroupInfo info = rsGroupMap.get(RSGroupInfo.DEFAULT_GROUP);
RSGroupInfo newInfo = new RSGroupInfo(info.getName(), servers);
HashMap<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
newGroupMap.put(newInfo.getName(), newInfo);
- flushConfig(newGroupMap);
+ resetRSGroupMap(newGroupMap);
}
/**
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
index a8cd277..28131a9 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
@@ -26,8 +26,11 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.SortedSet;
+import java.util.regex.Pattern;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
@@ -133,6 +136,13 @@ public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
tds.addAll(admin.listTableDescriptors());
tds.addAll(admin.listTableDescriptorsByNamespace(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME));
}
+ SortedSet<Address> lives = Sets.newTreeSet();
+ for (ServerName sn : conn.getAdmin().getClusterMetrics().getLiveServerMetrics().keySet()) {
+ lives.add(sn.getAddress());
+ }
+ for (ServerName sn : conn.getAdmin().listDecommissionedRegionServers()) {
+ lives.remove(sn.getAddress());
+ }
try (Table table = conn.getTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME);
ResultScanner scanner = table.getScanner(new Scan())) {
for (;;) {
@@ -144,8 +154,22 @@ public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
RSGroupInfoManagerImpl.META_FAMILY_BYTES, RSGroupInfoManagerImpl.META_QUALIFIER_BYTES));
RSGroupInfo rsGroupInfo = ProtobufUtil.toGroupInfo(proto);
groupMap.put(proto.getName(), RSGroupUtil.fillTables(rsGroupInfo, tds));
+ for(Address address : rsGroupInfo.getServers()){
+ lives.remove(address);
+ }
}
}
+ SortedSet<TableName> tables = Sets.newTreeSet();
+ for (TableDescriptor td : conn.getAdmin().listTableDescriptors(Pattern.compile(".*"),
+ true)){
+ String groupName = td.getRegionServerGroup().orElse(RSGroupInfo.DEFAULT_GROUP);
+ if (groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
+ tables.add(td.getTableName());
+ }
+ }
+
+ groupMap.put(RSGroupInfo.DEFAULT_GROUP,
+ new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, lives, tables));
assertEquals(Sets.newHashSet(groupMap.values()), Sets.newHashSet(wrapped.listRSGroups()));
try {
String groupBasePath = ZNodePaths.joinZNode(zkw.getZNodePaths().baseZNode, "rsgroup");
@@ -160,6 +184,7 @@ public class VerifyingRSGroupAdminClient extends RSGroupAdminClient {
zList.add(RSGroupUtil.fillTables(rsGroupInfo, tds));
}
}
+ groupMap.remove(RSGroupInfo.DEFAULT_GROUP);
assertEquals(zList.size(), groupMap.size());
for (RSGroupInfo rsGroupInfo : zList) {
assertTrue(groupMap.get(rsGroupInfo.getName()).equals(rsGroupInfo));
[hbase] 11/14: HBASE-23081 Add an option to enable/disable rs group
feature (#691)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 5d9620d4ea2f2ece71258da9fc5c37a211e66c13
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Mon Oct 7 17:34:33 2019 +0800
HBASE-23081 Add an option to enable/disable rs group feature (#691)
Signed-off-by: Peter Somogyi <ps...@apache.org>
---
.../org/apache/hadoop/hbase/master/HMaster.java | 12 +++
.../hbase/rsgroup/DisabledRSGroupInfoManager.java | 110 +++++++++++++++++++++
.../hadoop/hbase/rsgroup/RSGroupInfoManager.java | 8 +-
.../org/apache/hadoop/hbase/TestNamespace.java | 3 +-
.../hadoop/hbase/master/AbstractTestDLS.java | 4 +-
.../hadoop/hbase/master/TestClusterRestart.java | 2 +-
.../hadoop/hbase/master/TestMasterMetrics.java | 4 +-
.../TestMasterRestartAfterDisablingTable.java | 4 +-
.../hadoop/hbase/master/TestRollingRestart.java | 2 +-
.../hadoop/hbase/regionserver/TestRegionOpen.java | 1 -
.../TestRegionReplicasWithRestartScenarios.java | 2 +-
.../regionserver/TestRegionServerMetrics.java | 4 +-
.../hbase/util/TestHBaseFsckReplication.java | 2 -
13 files changed, 140 insertions(+), 18 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 04c48a6..8035953 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -186,6 +186,7 @@ import org.apache.hadoop.hbase.replication.master.ReplicationHFileCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationPeerConfigUpgrader;
import org.apache.hadoop.hbase.replication.regionserver.ReplicationStatus;
+import org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint;
import org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer;
import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.AccessDeniedException;
@@ -784,6 +785,17 @@ public class HMaster extends HRegionServer implements MasterServices {
this.splitOrMergeTracker = new SplitOrMergeTracker(zooKeeper, conf, this);
this.splitOrMergeTracker.start();
+ // This is for backwards compatible. We do not need the CP for rs group now but if user want to
+ // load it, we need to enable rs group.
+ String[] cpClasses = conf.getStrings(MasterCoprocessorHost.MASTER_COPROCESSOR_CONF_KEY);
+ if (cpClasses != null) {
+ for (String cpClass : cpClasses) {
+ if (RSGroupAdminEndpoint.class.getName().equals(cpClass)) {
+ conf.setBoolean(RSGroupInfoManager.RS_GROUP_ENABLED, true);
+ break;
+ }
+ }
+ }
this.rsGroupInfoManager = RSGroupInfoManager.create(this);
this.replicationPeerManager = ReplicationPeerManager.create(zooKeeper, conf);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java
new file mode 100644
index 0000000..c7c521c
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/DisabledRSGroupInfoManager.java
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.rsgroup;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.master.ServerManager;
+import org.apache.hadoop.hbase.net.Address;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * A dummy RSGroupInfoManager which only contains a default rs group.
+ */
+@InterfaceAudience.Private
+class DisabledRSGroupInfoManager implements RSGroupInfoManager {
+
+ private final ServerManager serverManager;
+
+ public DisabledRSGroupInfoManager(ServerManager serverManager) {
+ this.serverManager = serverManager;
+ }
+
+ @Override
+ public void start() {
+ }
+
+ @Override
+ public void addRSGroup(RSGroupInfo rsGroupInfo) throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
+
+ @Override
+ public void removeRSGroup(String groupName) throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
+
+ @Override
+ public Set<Address> moveServers(Set<Address> servers, String srcGroup, String dstGroup)
+ throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
+
+ private SortedSet<Address> getOnlineServers() {
+ SortedSet<Address> onlineServers = new TreeSet<Address>();
+ serverManager.getOnlineServers().keySet().stream().map(ServerName::getAddress)
+ .forEach(onlineServers::add);
+ return onlineServers;
+ }
+
+ @Override
+ public RSGroupInfo getRSGroupOfServer(Address serverHostPort) throws IOException {
+ SortedSet<Address> onlineServers = getOnlineServers();
+ if (onlineServers.contains(serverHostPort)) {
+ return new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, onlineServers);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public RSGroupInfo getRSGroup(String groupName) throws IOException {
+ if (RSGroupInfo.DEFAULT_GROUP.equals(groupName)) {
+ return new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, getOnlineServers());
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public List<RSGroupInfo> listRSGroups() throws IOException {
+ return Arrays.asList(new RSGroupInfo(RSGroupInfo.DEFAULT_GROUP, getOnlineServers()));
+ }
+
+ @Override
+ public boolean isOnline() {
+ return true;
+ }
+
+ @Override
+ public void removeServers(Set<Address> servers) throws IOException {
+ throw new DoNotRetryIOException("RSGroup is disabled");
+ }
+
+ @Override
+ public RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException {
+ return null;
+ }
+}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index a46fa4b..3f73f78 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -32,6 +32,8 @@ import org.apache.yetus.audience.InterfaceAudience;
@InterfaceAudience.Private
public interface RSGroupInfoManager {
+ static final String RS_GROUP_ENABLED = "hbase.balancer.rsgroup.enabled";
+
void start();
/**
@@ -90,6 +92,10 @@ public interface RSGroupInfoManager {
RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException;
static RSGroupInfoManager create(MasterServices master) throws IOException {
- return RSGroupInfoManagerImpl.getInstance(master);
+ if (master.getConfiguration().getBoolean(RS_GROUP_ENABLED, false)) {
+ return RSGroupInfoManagerImpl.getInstance(master);
+ } else {
+ return new DisabledRSGroupInfoManager(master.getServerManager());
+ }
}
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
index df39862..34ff391 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNamespace.java
@@ -114,8 +114,7 @@ public class TestNamespace {
assertEquals(2, admin.listNamespaceDescriptors().length);
//verify existence of system tables
- Set<TableName> systemTables = Sets.newHashSet(
- TableName.META_TABLE_NAME, TableName.valueOf("hbase:rsgroup"));
+ Set<TableName> systemTables = Sets.newHashSet(TableName.META_TABLE_NAME);
List<TableDescriptor> descs = admin.listTableDescriptorsByNamespace(
Bytes.toBytes(NamespaceDescriptor.SYSTEM_NAMESPACE.getName()));
assertEquals(systemTables.size(), descs.size());
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
index fc2e9f7..a576adc 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
@@ -504,14 +504,14 @@ public abstract class AbstractTestDLS {
for (String oregion : regions)
LOG.debug("Region still online: " + oregion);
}
- assertEquals(2 + existingRegions, regions.size());
+ assertEquals(1 + existingRegions, regions.size());
LOG.debug("Enabling table\n");
TEST_UTIL.getAdmin().enableTable(tableName);
LOG.debug("Waiting for no more RIT\n");
blockUntilNoRIT();
LOG.debug("Verifying there are " + numRegions + " assigned on cluster\n");
regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
- assertEquals(numRegions + 2 + existingRegions, regions.size());
+ assertEquals(numRegions + 1 + existingRegions, regions.size());
return table;
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
index a8d80b6..e880f97 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestart.java
@@ -43,7 +43,7 @@ public class TestClusterRestart extends AbstractTestRestartCluster {
private static final Logger LOG = LoggerFactory.getLogger(TestClusterRestart.class);
- private static final int NUM_REGIONS = 4;
+ private static final int NUM_REGIONS = 3;
@Override
protected boolean splitWALCoordinatedByZk() {
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
index 61720df..75d9ee1 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetrics.java
@@ -25,7 +25,6 @@ import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.MiniHBaseCluster;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.StartMiniClusterOption;
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.test.MetricsAssertHelper;
import org.apache.hadoop.hbase.testclassification.MasterTests;
import org.apache.hadoop.hbase.testclassification.MediumTests;
@@ -91,7 +90,6 @@ public class TestMasterMetrics {
cluster = TEST_UTIL.getHBaseCluster();
LOG.info("Waiting for active/ready master");
cluster.waitForActiveAndReadyMaster();
- TEST_UTIL.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
master = cluster.getMaster();
}
@@ -133,7 +131,7 @@ public class TestMasterMetrics {
MetricsMasterSource masterSource = master.getMasterMetrics().getMetricsSource();
boolean tablesOnMaster = LoadBalancer.isTablesOnMaster(TEST_UTIL.getConfiguration());
metricsHelper.assertGauge("numRegionServers", 1 + (tablesOnMaster ? 1 : 0), masterSource);
- metricsHelper.assertGauge("averageLoad", 2, masterSource);
+ metricsHelper.assertGauge("averageLoad", 1, masterSource);
metricsHelper.assertGauge("numDeadRegionServers", 0, masterSource);
metricsHelper.assertGauge("masterStartTime", master.getMasterStartTime(), masterSource);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
index 2d0af3e..fca2866 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
@@ -92,7 +92,7 @@ public class TestMasterRestartAfterDisablingTable {
NavigableSet<String> regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
assertEquals("The number of regions for the table tableRestart should be 0 and only" +
- "the catalog table should be present.", 2, regions.size());
+ "the catalog table should be present.", 1, regions.size());
List<MasterThread> masterThreads = cluster.getMasterThreads();
MasterThread activeMaster = null;
@@ -120,7 +120,7 @@ public class TestMasterRestartAfterDisablingTable {
log("Verifying there are " + numRegions + " assigned on cluster\n");
regions = HBaseTestingUtility.getAllOnlineRegions(cluster);
assertEquals("The assigned regions were not onlined after master" +
- " switch except for the catalog table.", 6, regions.size());
+ " switch except for the catalog table.", 5, regions.size());
assertTrue("The table should be in enabled state", cluster.getMaster().getTableStateManager()
.isTableState(TableName.valueOf(name.getMethodName()), TableState.State.ENABLED));
ht.close();
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
index 2dc1b6c..0aba487 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
@@ -121,7 +121,7 @@ public class TestRollingRestart {
log("Region still online: " + oregion);
}
}
- assertEquals(2, regions.size());
+ assertEquals(1, regions.size());
log("Enabling table\n");
TEST_UTIL.getAdmin().enableTable(tableName);
log("Waiting for no more RIT\n");
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
index 2503825..7d52508 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
@@ -86,7 +86,6 @@ public class TestRegionOpen {
final TableName tableName = TableName.valueOf(TestRegionOpen.class.getSimpleName());
ThreadPoolExecutor exec = getRS().getExecutorService()
.getExecutorThreadPool(ExecutorType.RS_OPEN_PRIORITY_REGION);
- HTU.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
long completed = exec.getCompletedTaskCount();
HTableDescriptor htd = new HTableDescriptor(tableName);
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
index dc0a918..70df61e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
@@ -146,7 +146,7 @@ public class TestRegionReplicasWithRestartScenarios {
checkDuplicates(onlineRegions3);
assertFalse(res);
int totalRegions = onlineRegions.size() + onlineRegions2.size() + onlineRegions3.size();
- assertEquals(62, totalRegions);
+ assertEquals(61, totalRegions);
}
private boolean checkDuplicates(Collection<HRegion> onlineRegions3) throws Exception {
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
index 952bcbc..bc44daa 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
@@ -225,7 +225,7 @@ public class TestRegionServerMetrics {
@Test
public void testRegionCount() throws Exception {
- metricsHelper.assertGauge("regionCount", TABLES_ON_MASTER ? 1 : 3, serverSource);
+ metricsHelper.assertGauge("regionCount", TABLES_ON_MASTER ? 1 : 2, serverSource);
}
@Test
@@ -335,7 +335,7 @@ public class TestRegionServerMetrics {
TEST_UTIL.getAdmin().flush(tableName);
metricsRegionServer.getRegionServerWrapper().forceRecompute();
- assertGauge("storeCount", TABLES_ON_MASTER ? 1 : 6);
+ assertGauge("storeCount", TABLES_ON_MASTER ? 1 : 5);
assertGauge("storeFileCount", 1);
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
index 0ac07d8..316a321 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplication.java
@@ -24,7 +24,6 @@ import java.util.stream.Stream;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.ServerName;
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerStorage;
import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
@@ -53,7 +52,6 @@ public class TestHBaseFsckReplication {
public static void setUp() throws Exception {
UTIL.getConfiguration().setBoolean("hbase.write.hbck1.lock.file", false);
UTIL.startMiniCluster(1);
- UTIL.waitTableAvailable(TableName.valueOf("hbase:rsgroup"));
}
@AfterClass
[hbase] 08/14: HBASE-22729 Start RSGroupInfoManager as default
(#555)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 90d4fb7082bf1fede4aecdcb3d9304d78ea1eac3
Author: Guanghao Zhang <zg...@apache.org>
AuthorDate: Tue Sep 3 11:39:19 2019 +0800
HBASE-22729 Start RSGroupInfoManager as default (#555)
Amending-Author: Duo Zhang <zh...@apache.org>
Signed-off-by: stack <st...@apache.org>
---
.../src/main/java/org/apache/hadoop/hbase/master/HMaster.java | 9 +++++++++
.../main/java/org/apache/hadoop/hbase/master/MasterServices.java | 7 +++++--
.../org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 2 +-
.../java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java | 5 +++++
.../org/apache/hadoop/hbase/master/MockNoopMasterServices.java | 6 ++++++
5 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 3a33e5f..3e0e18d 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -188,6 +188,7 @@ import org.apache.hadoop.hbase.replication.master.ReplicationHFileCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner;
import org.apache.hadoop.hbase.replication.master.ReplicationPeerConfigUpgrader;
import org.apache.hadoop.hbase.replication.regionserver.ReplicationStatus;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.AccessDeniedException;
import org.apache.hadoop.hbase.security.SecurityConstants;
import org.apache.hadoop.hbase.security.UserProvider;
@@ -355,6 +356,8 @@ public class HMaster extends HRegionServer implements MasterServices {
// manager of assignment nodes in zookeeper
private AssignmentManager assignmentManager;
+ private RSGroupInfoManager rsGroupInfoManager;
+
// manager of replication
private ReplicationPeerManager replicationPeerManager;
@@ -784,6 +787,8 @@ public class HMaster extends HRegionServer implements MasterServices {
this.splitOrMergeTracker = new SplitOrMergeTracker(zooKeeper, conf, this);
this.splitOrMergeTracker.start();
+ this.rsGroupInfoManager = RSGroupInfoManager.create(this);
+
this.replicationPeerManager = ReplicationPeerManager.create(zooKeeper, conf);
this.drainingServerTracker = new DrainingServerTracker(zooKeeper, this, this.serverManager);
@@ -3869,4 +3874,8 @@ public class HMaster extends HRegionServer implements MasterServices {
return cachedClusterId.getFromCacheOrFetch();
}
+ @Override
+ public RSGroupInfoManager getRSRSGroupInfoManager() {
+ return rsGroupInfoManager;
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
index 41cec5c..ead40b9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
@@ -18,10 +18,8 @@
package org.apache.hadoop.hbase.master;
import com.google.protobuf.Service;
-
import java.io.IOException;
import java.util.List;
-
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableDescriptors;
@@ -51,6 +49,7 @@ import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.hadoop.hbase.security.access.ZKPermissionWatcher;
import org.apache.yetus.audience.InterfaceAudience;
@@ -537,4 +536,8 @@ public interface MasterServices extends Server {
*/
List<RegionPlan> executeRegionPlansWithThrottling(List<RegionPlan> plans);
+ /**
+ * @return the {@link RSGroupInfoManager}
+ */
+ RSGroupInfoManager getRSRSGroupInfoManager();
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index a2a5623..0bde67b 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -66,7 +66,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
}
master = ((HasMasterServices) env).getMasterServices();
- groupInfoManager = RSGroupInfoManagerImpl.getInstance(master);
+ groupInfoManager = master.getRSRSGroupInfoManager();
groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
Class<?> clazz =
master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 1b9f3ef..a46fa4b 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.List;
import java.util.Set;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -87,4 +88,8 @@ public interface RSGroupInfoManager {
*/
@Deprecated
RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException;
+
+ static RSGroupInfoManager create(MasterServices master) throws IOException {
+ return RSGroupInfoManagerImpl.getInstance(master);
+ }
}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
index cbfdd3f..26772ae 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java
@@ -54,6 +54,7 @@ import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
import org.apache.hadoop.hbase.replication.ReplicationPeerDescription;
import org.apache.hadoop.hbase.replication.SyncReplicationState;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfoManager;
import org.apache.hadoop.hbase.security.access.AccessChecker;
import org.apache.hadoop.hbase.security.access.ZKPermissionWatcher;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
@@ -490,4 +491,9 @@ public class MockNoopMasterServices implements MasterServices {
public AsyncClusterConnection getAsyncClusterConnection() {
return null;
}
+
+ @Override
+ public RSGroupInfoManager getRSRSGroupInfoManager() {
+ return null;
+ }
}
\ No newline at end of file
[hbase] 02/14: HBASE-22662 Move RSGroupInfoManager to hbase-server
(#368)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit da6250690200e84459e61d80d29a44aef670259b
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Thu Jul 11 10:34:05 2019 +0800
HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
hbase-rsgroup/pom.xml | 4 -
.../hbase/rsgroup/RSGroupBasedLoadBalancer.java | 4 +-
.../hbase/rsgroup/TestRSGroupsOfflineMode.java | 6 +-
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 6 +-
.../hadoop/hbase/rsgroup/RSGroupInfoManager.java | 26 +----
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 108 +++++++++++----------
6 files changed, 71 insertions(+), 83 deletions(-)
diff --git a/hbase-rsgroup/pom.xml b/hbase-rsgroup/pom.xml
index 09c8e73..1135939 100644
--- a/hbase-rsgroup/pom.xml
+++ b/hbase-rsgroup/pom.xml
@@ -113,10 +113,6 @@
<artifactId>hbase-shaded-miscellaneous</artifactId>
</dependency>
<dependency>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </dependency>
- <dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</dependency>
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index 6767ac9..7394f55 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++ b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -121,8 +121,8 @@ public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
public List<RegionPlan> balanceCluster(Map<ServerName, List<RegionInfo>> clusterState)
throws HBaseIOException {
if (!isOnline()) {
- throw new ConstraintException(RSGroupInfoManager.RSGROUP_TABLE_NAME +
- " is not online, unable to perform balance");
+ throw new ConstraintException(
+ RSGroupInfoManager.class.getSimpleName() + " is not online, unable to perform balance");
}
// Calculate correct assignments and a list of RegionPlan for mis-placed regions
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
index 39cf164..60887e4 100644
--- a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
+++ b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
@@ -140,7 +140,7 @@ public class TestRSGroupsOfflineMode {
}
});
// Move table to group and wait.
- groupAdmin.moveTables(Sets.newHashSet(RSGroupInfoManager.RSGROUP_TABLE_NAME), newGroup);
+ groupAdmin.moveTables(Sets.newHashSet(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME), newGroup);
LOG.info("Waiting for move table...");
TEST_UTIL.waitFor(WAIT_TIMEOUT, new Waiter.Predicate<Exception>() {
@Override
@@ -169,7 +169,7 @@ public class TestRSGroupsOfflineMode {
// Make sure balancer is in offline mode, since this is what we're testing.
assertFalse(groupMgr.isOnline());
// Verify the group affiliation that's loaded from ZK instead of tables.
- assertEquals(newGroup, groupMgr.getRSGroupOfTable(RSGroupInfoManager.RSGROUP_TABLE_NAME));
+ assertEquals(newGroup, groupMgr.getRSGroupOfTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME));
assertEquals(RSGroupInfo.DEFAULT_GROUP, groupMgr.getRSGroupOfTable(failoverTable));
// Kill final regionserver to see the failover happens for all tables except GROUP table since
// it's group does not have any online RS.
@@ -182,7 +182,7 @@ public class TestRSGroupsOfflineMode {
return failoverRS.getRegions(failoverTable).size() >= 1;
}
});
- Assert.assertEquals(0, failoverRS.getRegions(RSGroupInfoManager.RSGROUP_TABLE_NAME).size());
+ Assert.assertEquals(0, failoverRS.getRegions(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME).size());
// Need this for minicluster to shutdown cleanly.
master.stopMaster();
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
index 2ad30e4..fcaf1a7 100644
--- a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
+++ b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
@@ -52,7 +52,7 @@ public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
throws IOException {
wrapped = RSGroupAdmin;
table = ConnectionFactory.createConnection(conf)
- .getTable(RSGroupInfoManager.RSGROUP_TABLE_NAME);
+ .getTable(RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME);
zkw = new ZKWatcher(conf, this.getClass().getSimpleName(), null);
}
@@ -126,8 +126,8 @@ public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
RSGroupProtos.RSGroupInfo proto =
RSGroupProtos.RSGroupInfo.parseFrom(
result.getValue(
- RSGroupInfoManager.META_FAMILY_BYTES,
- RSGroupInfoManager.META_QUALIFIER_BYTES));
+ RSGroupInfoManagerImpl.META_FAMILY_BYTES,
+ RSGroupInfoManagerImpl.META_QUALIFIER_BYTES));
groupMap.put(proto.getName(), ProtobufUtil.toGroupInfo(proto));
}
Assert.assertEquals(Sets.newHashSet(groupMap.values()),
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
similarity index 82%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 398e8a4..70aeabf 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -15,38 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
import java.util.List;
import java.util.Set;
-
-import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.net.Address;
-import org.apache.hadoop.hbase.util.Bytes;
import org.apache.yetus.audience.InterfaceAudience;
/**
- * Interface used to manage RSGroupInfo storage. An implementation
- * has the option to support offline mode.
- * See {@link RSGroupBasedLoadBalancer}
+ * Interface used to manage RSGroupInfo storage. An implementation has the option to support offline
+ * mode. See {@code RSGroupBasedLoadBalancer}.
*/
@InterfaceAudience.Private
public interface RSGroupInfoManager {
- String REASSIGN_WAIT_INTERVAL_KEY = "hbase.rsgroup.reassign.wait";
- long DEFAULT_REASSIGN_WAIT_INTERVAL = 30 * 1000L;
-
- //Assigned before user tables
- TableName RSGROUP_TABLE_NAME =
- TableName.valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, "rsgroup");
- String rsGroupZNode = "rsgroup";
- byte[] META_FAMILY_BYTES = Bytes.toBytes("m");
- byte[] META_QUALIFIER_BYTES = Bytes.toBytes("i");
- byte[] ROW_KEY = {0};
-
void start();
/**
@@ -86,7 +70,6 @@ public interface RSGroupInfoManager {
/**
* Set the group membership of a set of tables
- *
* @param tableNames set of tables to move
* @param groupName name of group of tables to move to
*/
@@ -104,7 +87,6 @@ public interface RSGroupInfoManager {
/**
* Whether the manager is able to fully return group metadata
- *
* @return whether the manager is in online mode
*/
boolean isOnline();
@@ -116,8 +98,8 @@ public interface RSGroupInfoManager {
* @param srcGroup groupName being moved from
* @param dstGroup groupName being moved to
*/
- void moveServersAndTables(Set<Address> servers, Set<TableName> tables,
- String srcGroup, String dstGroup) throws IOException;
+ void moveServersAndTables(Set<Address> servers, Set<TableName> tables, String srcGroup,
+ String dstGroup) throws IOException;
/**
* Remove decommissioned servers from rsgroup
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
similarity index 90%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index b54f088..8aa7520 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hbase.rsgroup;
-import com.google.protobuf.ServiceException;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
@@ -34,10 +33,12 @@ import java.util.SortedSet;
import java.util.TreeSet;
import org.apache.hadoop.hbase.Coprocessor;
import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.AsyncTable;
import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
-import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.CoprocessorDescriptorBuilder;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
@@ -45,14 +46,11 @@ import org.apache.hadoop.hbase.client.Mutation;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
-import org.apache.hadoop.hbase.client.Scan;
-import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.constraint.ConstraintException;
import org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
-import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
import org.apache.hadoop.hbase.master.MasterServices;
import org.apache.hadoop.hbase.master.ServerListener;
import org.apache.hadoop.hbase.master.TableStateManager;
@@ -62,10 +60,14 @@ import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.procedure2.Procedure;
import org.apache.hadoop.hbase.protobuf.ProtobufMagic;
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
-import org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos;
+import org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto;
+import org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos.MultiRowMutationService;
+import org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos.MutateRowsRequest;
+import org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos.MutateRowsResponse;
import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
import org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy;
import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.FutureUtils;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
@@ -75,6 +77,7 @@ import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
@@ -87,13 +90,13 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
* RSGroupInfo Map at {@link #rsGroupMap} and a Map of tables to the name of the rsgroup they belong
* too (in {@link #tableMap}). These Maps are persisted to the hbase:rsgroup table (and cached in
* zk) on each modification.
- * <p>
+ * <p/>
* Mutations on state are synchronized but reads can continue without having to wait on an instance
* monitor, mutations do wholesale replace of the Maps on update -- Copy-On-Write; the local Maps of
* state are read-only, just-in-case (see flushConfig).
- * <p>
+ * <p/>
* Reads must not block else there is a danger we'll deadlock.
- * <p>
+ * <p/>
* Clients of this class, the {@link RSGroupAdminEndpoint} for example, want to query and then act
* on the results of the query modifying cache in zookeeper without another thread making
* intermediate modifications. These clients synchronize on the 'this' instance so no other has
@@ -103,6 +106,24 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private static final Logger LOG = LoggerFactory.getLogger(RSGroupInfoManagerImpl.class);
+ private static final String REASSIGN_WAIT_INTERVAL_KEY = "hbase.rsgroup.reassign.wait";
+ private static final long DEFAULT_REASSIGN_WAIT_INTERVAL = 30 * 1000L;
+
+ // Assigned before user tables
+ @VisibleForTesting
+ static final TableName RSGROUP_TABLE_NAME =
+ TableName.valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, "rsgroup");
+
+ private static final String RS_GROUP_ZNODE = "rsgroup";
+
+ @VisibleForTesting
+ static final byte[] META_FAMILY_BYTES = Bytes.toBytes("m");
+
+ @VisibleForTesting
+ static final byte[] META_QUALIFIER_BYTES = Bytes.toBytes("i");
+
+ private static final byte[] ROW_KEY = { 0 };
+
/** Table descriptor for <code>hbase:rsgroup</code> catalog table */
private static final TableDescriptor RSGROUP_TABLE_DESC;
static {
@@ -125,7 +146,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private volatile Map<TableName, String> tableMap = Collections.emptyMap();
private final MasterServices masterServices;
- private final Connection conn;
+ private final AsyncClusterConnection conn;
private final ZKWatcher watcher;
private final RSGroupStartupWorker rsGroupStartupWorker;
// contains list of groups that were last flushed to persistent store
@@ -136,7 +157,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private RSGroupInfoManagerImpl(MasterServices masterServices) throws IOException {
this.masterServices = masterServices;
this.watcher = masterServices.getZooKeeper();
- this.conn = masterServices.getConnection();
+ this.conn = masterServices.getAsyncClusterConnection();
this.rsGroupStartupWorker = new RSGroupStartupWorker();
}
@@ -349,25 +370,25 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
}
- List<RSGroupInfo> retrieveGroupListFromGroupTable() throws IOException {
+ private List<RSGroupInfo> retrieveGroupListFromGroupTable() throws IOException {
List<RSGroupInfo> rsGroupInfoList = Lists.newArrayList();
- try (Table table = conn.getTable(RSGROUP_TABLE_NAME);
- ResultScanner scanner = table.getScanner(new Scan())) {
+ AsyncTable<?> table = conn.getTable(RSGROUP_TABLE_NAME);
+ try (ResultScanner scanner = table.getScanner(META_FAMILY_BYTES, META_QUALIFIER_BYTES)) {
for (Result result;;) {
result = scanner.next();
if (result == null) {
break;
}
RSGroupProtos.RSGroupInfo proto = RSGroupProtos.RSGroupInfo
- .parseFrom(result.getValue(META_FAMILY_BYTES, META_QUALIFIER_BYTES));
+ .parseFrom(result.getValue(META_FAMILY_BYTES, META_QUALIFIER_BYTES));
rsGroupInfoList.add(ProtobufUtil.toGroupInfo(proto));
}
}
return rsGroupInfoList;
}
- List<RSGroupInfo> retrieveGroupListFromZookeeper() throws IOException {
- String groupBasePath = ZNodePaths.joinZNode(watcher.getZNodePaths().baseZNode, rsGroupZNode);
+ private List<RSGroupInfo> retrieveGroupListFromZookeeper() throws IOException {
+ String groupBasePath = ZNodePaths.joinZNode(watcher.getZNodePaths().baseZNode, RS_GROUP_ZNODE);
List<RSGroupInfo> RSGroupInfoList = Lists.newArrayList();
// Overwrite any info stored by table, this takes precedence
try {
@@ -519,7 +540,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
resetRSGroupAndTableMaps(newGroupMap, newTableMap);
try {
- String groupBasePath = ZNodePaths.joinZNode(watcher.getZNodePaths().baseZNode, rsGroupZNode);
+ String groupBasePath =
+ ZNodePaths.joinZNode(watcher.getZNodePaths().baseZNode, RS_GROUP_ZNODE);
ZKUtil.createAndFailSilent(watcher, groupBasePath, ProtobufMagic.PB_MAGIC);
List<ZKUtil.ZKUtilOp> zkOps = new ArrayList<>(newGroupMap.size());
@@ -702,11 +724,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
createRSGroupTable();
}
// try reading from the table
- try (Table table = conn.getTable(RSGROUP_TABLE_NAME)) {
- table.get(new Get(ROW_KEY));
- }
- LOG.info(
- "RSGroup table=" + RSGROUP_TABLE_NAME + " is online, refreshing cached information");
+ FutureUtils.get(conn.getTable(RSGROUP_TABLE_NAME).get(new Get(ROW_KEY)));
+ LOG.info("RSGroup table={} is online, refreshing cached information", RSGROUP_TABLE_NAME);
RSGroupInfoManagerImpl.this.refresh(true);
online = true;
// flush any inconsistencies between ZK and HTable
@@ -748,8 +767,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
} else {
Procedure<?> result = masterServices.getMasterProcedureExecutor().getResult(procId);
if (result != null && result.isFailed()) {
- throw new IOException(
- "Failed to create group table. " + MasterProcedureUtil.unwrapRemoteIOException(result));
+ throw new IOException("Failed to create group table. " +
+ MasterProcedureUtil.unwrapRemoteIOException(result));
}
}
}
@@ -764,33 +783,24 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
private void multiMutate(List<Mutation> mutations) throws IOException {
- try (Table table = conn.getTable(RSGROUP_TABLE_NAME)) {
- CoprocessorRpcChannel channel = table.coprocessorService(ROW_KEY);
- MultiRowMutationProtos.MutateRowsRequest.Builder mmrBuilder =
- MultiRowMutationProtos.MutateRowsRequest.newBuilder();
- for (Mutation mutation : mutations) {
- if (mutation instanceof Put) {
- mmrBuilder.addMutationRequest(org.apache.hadoop.hbase.protobuf.ProtobufUtil.toMutation(
- org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType.PUT,
- mutation));
- } else if (mutation instanceof Delete) {
- mmrBuilder.addMutationRequest(org.apache.hadoop.hbase.protobuf.ProtobufUtil.toMutation(
- org.apache.hadoop.hbase.protobuf.generated.ClientProtos.MutationProto.MutationType.DELETE,
- mutation));
- } else {
- throw new DoNotRetryIOException(
+ MutateRowsRequest.Builder builder = MutateRowsRequest.newBuilder();
+ for (Mutation mutation : mutations) {
+ if (mutation instanceof Put) {
+ builder
+ .addMutationRequest(ProtobufUtil.toMutation(MutationProto.MutationType.PUT, mutation));
+ } else if (mutation instanceof Delete) {
+ builder.addMutationRequest(
+ ProtobufUtil.toMutation(MutationProto.MutationType.DELETE, mutation));
+ } else {
+ throw new DoNotRetryIOException(
"multiMutate doesn't support " + mutation.getClass().getName());
- }
- }
-
- MultiRowMutationProtos.MultiRowMutationService.BlockingInterface service =
- MultiRowMutationProtos.MultiRowMutationService.newBlockingStub(channel);
- try {
- service.mutateRows(null, mmrBuilder.build());
- } catch (ServiceException ex) {
- ProtobufUtil.toIOException(ex);
}
}
+ MutateRowsRequest request = builder.build();
+ AsyncTable<?> table = conn.getTable(RSGROUP_TABLE_NAME);
+ FutureUtils.get(table.<MultiRowMutationService, MutateRowsResponse> coprocessorService(
+ MultiRowMutationService::newStub,
+ (stub, controller, done) -> stub.mutateRows(controller, request, done), ROW_KEY));
}
private void checkGroupName(String groupName) throws ConstraintException {
[hbase] 01/14: HBASE-22664 Move protobuf stuff in hbase-rsgroup to
hbase-protocol-shaded (#362)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 13df3a03cc5b8efc530b4bcd3dcfb547f78d567a
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Tue Jul 9 09:51:19 2019 +0800
HBASE-22664 Move protobuf stuff in hbase-rsgroup to hbase-protocol-shaded (#362)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../apache/hadoop/hbase/protobuf/ProtobufUtil.java | 29 +++++++---
.../hadoop/hbase/shaded/protobuf/ProtobufUtil.java | 27 ++++++++++
.../src/main/protobuf/RSGroup.proto | 33 ++++++++++++
.../src/main/protobuf/RSGroupAdmin.proto | 2 +-
.../src/main/protobuf/RSGroupAdmin.proto | 0
hbase-rsgroup/pom.xml | 18 -------
.../hadoop/hbase/rsgroup/RSGroupAdminClient.java | 10 ++--
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 10 ++--
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 8 +--
.../hadoop/hbase/rsgroup/RSGroupProtobufUtil.java | 63 ----------------------
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 5 +-
11 files changed, 96 insertions(+), 109 deletions(-)
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
index a73a2bc..6f72004 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
@@ -1772,23 +1772,36 @@ public final class ProtobufUtil {
return ServerName.valueOf(hostname, port, -1L);
}
+ public static HBaseProtos.TimeRange toTimeRange(TimeRange timeRange) {
+ if (timeRange == null) {
+ timeRange = TimeRange.allTime();
+ }
+ return HBaseProtos.TimeRange.newBuilder().setFrom(timeRange.getMin()).setTo(timeRange.getMax())
+ .build();
+ }
+
public static RSGroupInfo toGroupInfo(RSGroupProtos.RSGroupInfo proto) {
RSGroupInfo RSGroupInfo = new RSGroupInfo(proto.getName());
- for(HBaseProtos.ServerName el: proto.getServersList()) {
+ for (HBaseProtos.ServerName el : proto.getServersList()) {
RSGroupInfo.addServer(Address.fromParts(el.getHostName(), el.getPort()));
}
- for(HBaseProtos.TableName pTableName: proto.getTablesList()) {
+ for (HBaseProtos.TableName pTableName : proto.getTablesList()) {
RSGroupInfo.addTable(ProtobufUtil.toTableName(pTableName));
}
return RSGroupInfo;
}
- public static HBaseProtos.TimeRange toTimeRange(TimeRange timeRange) {
- if (timeRange == null) {
- timeRange = TimeRange.allTime();
+ public static RSGroupProtos.RSGroupInfo toProtoGroupInfo(RSGroupInfo pojo) {
+ List<HBaseProtos.TableName> tables = new ArrayList<>(pojo.getTables().size());
+ for (TableName arg : pojo.getTables()) {
+ tables.add(ProtobufUtil.toProtoTableName(arg));
+ }
+ List<HBaseProtos.ServerName> hostports = new ArrayList<>(pojo.getServers().size());
+ for (Address el : pojo.getServers()) {
+ hostports.add(HBaseProtos.ServerName.newBuilder().setHostName(el.getHostname())
+ .setPort(el.getPort()).build());
}
- return HBaseProtos.TimeRange.newBuilder().setFrom(timeRange.getMin())
- .setTo(timeRange.getMax())
- .build();
+ return RSGroupProtos.RSGroupInfo.newBuilder().setName(pojo.getName()).addAllServers(hostports)
+ .addAllTables(tables).build();
}
}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
index 8108217..24b8dde 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
@@ -92,6 +92,7 @@ import org.apache.hadoop.hbase.exceptions.DeserializationException;
import org.apache.hadoop.hbase.filter.ByteArrayComparable;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.io.TimeRange;
+import org.apache.hadoop.hbase.net.Address;
import org.apache.hadoop.hbase.protobuf.ProtobufMagic;
import org.apache.hadoop.hbase.protobuf.ProtobufMessageConverter;
import org.apache.hadoop.hbase.quotas.QuotaScope;
@@ -100,6 +101,7 @@ import org.apache.hadoop.hbase.quotas.SpaceViolationPolicy;
import org.apache.hadoop.hbase.quotas.ThrottleType;
import org.apache.hadoop.hbase.replication.ReplicationLoadSink;
import org.apache.hadoop.hbase.replication.ReplicationLoadSource;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.security.visibility.Authorizations;
import org.apache.hadoop.hbase.security.visibility.CellVisibility;
import org.apache.hadoop.hbase.util.Addressing;
@@ -176,6 +178,7 @@ import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ListTableD
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MajorCompactionTimestampResponse;
import org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos;
import org.apache.hadoop.hbase.shaded.protobuf.generated.QuotaProtos;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.RSGroupProtos;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerReportRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos.RegionServerStartupRequest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos;
@@ -3348,4 +3351,28 @@ public final class ProtobufUtil {
.build();
}
+ public static RSGroupInfo toGroupInfo(RSGroupProtos.RSGroupInfo proto) {
+ RSGroupInfo RSGroupInfo = new RSGroupInfo(proto.getName());
+ for (HBaseProtos.ServerName el : proto.getServersList()) {
+ RSGroupInfo.addServer(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ for (HBaseProtos.TableName pTableName : proto.getTablesList()) {
+ RSGroupInfo.addTable(ProtobufUtil.toTableName(pTableName));
+ }
+ return RSGroupInfo;
+ }
+
+ public static RSGroupProtos.RSGroupInfo toProtoGroupInfo(RSGroupInfo pojo) {
+ List<HBaseProtos.TableName> tables = new ArrayList<>(pojo.getTables().size());
+ for (TableName arg : pojo.getTables()) {
+ tables.add(ProtobufUtil.toProtoTableName(arg));
+ }
+ List<HBaseProtos.ServerName> hostports = new ArrayList<>(pojo.getServers().size());
+ for (Address el : pojo.getServers()) {
+ hostports.add(HBaseProtos.ServerName.newBuilder().setHostName(el.getHostname())
+ .setPort(el.getPort()).build());
+ }
+ return RSGroupProtos.RSGroupInfo.newBuilder().setName(pojo.getName()).addAllServers(hostports)
+ .addAllTables(tables).build();
+ }
}
diff --git a/hbase-protocol-shaded/src/main/protobuf/RSGroup.proto b/hbase-protocol-shaded/src/main/protobuf/RSGroup.proto
new file mode 100644
index 0000000..ede2b13
--- /dev/null
+++ b/hbase-protocol-shaded/src/main/protobuf/RSGroup.proto
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package hbase.pb;
+
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "RSGroupProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
+
+import "HBase.proto";
+
+message RSGroupInfo {
+ required string name = 1;
+ repeated ServerName servers = 4;
+ repeated TableName tables = 3;
+}
diff --git a/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto b/hbase-protocol-shaded/src/main/protobuf/RSGroupAdmin.proto
similarity index 98%
copy from hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
copy to hbase-protocol-shaded/src/main/protobuf/RSGroupAdmin.proto
index 416097b..1db7136 100644
--- a/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
+++ b/hbase-protocol-shaded/src/main/protobuf/RSGroupAdmin.proto
@@ -18,7 +18,7 @@
package hbase.pb;
-option java_package = "org.apache.hadoop.hbase.protobuf.generated";
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
option java_outer_classname = "RSGroupAdminProtos";
option java_generic_services = true;
option java_generate_equals_and_hash = true;
diff --git a/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto b/hbase-protocol/src/main/protobuf/RSGroupAdmin.proto
similarity index 100%
rename from hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
rename to hbase-protocol/src/main/protobuf/RSGroupAdmin.proto
diff --git a/hbase-rsgroup/pom.xml b/hbase-rsgroup/pom.xml
index 0bef660..09c8e73 100644
--- a/hbase-rsgroup/pom.xml
+++ b/hbase-rsgroup/pom.xml
@@ -44,24 +44,6 @@
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
- <groupId>org.xolstice.maven.plugins</groupId>
- <artifactId>protobuf-maven-plugin</artifactId>
- <executions>
- <execution>
- <id>compile-protoc</id>
- <phase>generate-sources</phase>
- <goals>
- <goal>compile</goal>
- </goals>
- <configuration>
- <additionalProtoPathElements>
- <additionalProtoPathElement>${basedir}/../hbase-protocol/src/main/protobuf</additionalProtoPathElement>
- </additionalProtoPathElements>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
index e8a1410..e7ab7f2 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
+++ b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
@@ -18,12 +18,10 @@
package org.apache.hadoop.hbase.rsgroup;
import com.google.protobuf.ServiceException;
-
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
-
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.TableNotFoundException;
import org.apache.hadoop.hbase.client.Admin;
@@ -70,7 +68,7 @@ public class RSGroupAdminClient implements RSGroupAdmin {
GetRSGroupInfoResponse resp = stub.getRSGroupInfo(null,
GetRSGroupInfoRequest.newBuilder().setRSGroupName(groupName).build());
if(resp.hasRSGroupInfo()) {
- return RSGroupProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
+ return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
return null;
} catch (ServiceException e) {
@@ -85,7 +83,7 @@ public class RSGroupAdminClient implements RSGroupAdmin {
try {
GetRSGroupInfoOfTableResponse resp = stub.getRSGroupInfoOfTable(null, request);
if (resp.hasRSGroupInfo()) {
- return RSGroupProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
+ return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
return null;
} catch (ServiceException e) {
@@ -167,7 +165,7 @@ public class RSGroupAdminClient implements RSGroupAdmin {
ListRSGroupInfosRequest.getDefaultInstance()).getRSGroupInfoList();
List<RSGroupInfo> result = new ArrayList<>(resp.size());
for(RSGroupProtos.RSGroupInfo entry : resp) {
- result.add(RSGroupProtobufUtil.toGroupInfo(entry));
+ result.add(ProtobufUtil.toGroupInfo(entry));
}
return result;
} catch (ServiceException e) {
@@ -186,7 +184,7 @@ public class RSGroupAdminClient implements RSGroupAdmin {
try {
GetRSGroupInfoOfServerResponse resp = stub.getRSGroupInfoOfServer(null, request);
if (resp.hasRSGroupInfo()) {
- return RSGroupProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
+ return ProtobufUtil.toGroupInfo(resp.getRSGroupInfo());
}
return null;
} catch (ServiceException e) {
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
index 090ac6e..8c6e010 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -21,7 +21,6 @@ package org.apache.hadoop.hbase.rsgroup;
import com.google.protobuf.RpcCallback;
import com.google.protobuf.RpcController;
import com.google.protobuf.Service;
-
import java.io.IOException;
import java.util.Collections;
import java.util.HashSet;
@@ -29,7 +28,6 @@ import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
-
import org.apache.hadoop.hbase.CoprocessorEnvironment;
import org.apache.hadoop.hbase.HBaseIOException;
import org.apache.hadoop.hbase.HConstants;
@@ -164,7 +162,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
checkPermission("getRSGroupInfo");
RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
if (rsGroupInfo != null) {
- builder.setRSGroupInfo(RSGroupProtobufUtil.toProtoGroupInfo(rsGroupInfo));
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfo(groupName);
@@ -189,7 +187,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
checkPermission("getRSGroupInfoOfTable");
RSGroupInfo RSGroupInfo = groupAdminServer.getRSGroupInfoOfTable(tableName);
if (RSGroupInfo != null) {
- builder.setRSGroupInfo(RSGroupProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfoOfTable(tableName);
@@ -326,7 +324,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
}
checkPermission("listRSGroup");
for (RSGroupInfo RSGroupInfo : groupAdminServer.listRSGroups()) {
- builder.addRSGroupInfo(RSGroupProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postListRSGroups();
@@ -352,7 +350,7 @@ public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
checkPermission("getRSGroupInfoOfServer");
RSGroupInfo info = groupAdminServer.getRSGroupOfServer(hp);
if (info != null) {
- builder.setRSGroupInfo(RSGroupProtobufUtil.toProtoGroupInfo(info));
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(info));
}
if (master.getMasterCoprocessorHost() != null) {
master.getMasterCoprocessorHost().postGetRSGroupInfoOfServer(hp);
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index b2d168a..b54f088 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -360,7 +360,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
RSGroupProtos.RSGroupInfo proto = RSGroupProtos.RSGroupInfo
.parseFrom(result.getValue(META_FAMILY_BYTES, META_QUALIFIER_BYTES));
- rsGroupInfoList.add(RSGroupProtobufUtil.toGroupInfo(proto));
+ rsGroupInfoList.add(ProtobufUtil.toGroupInfo(proto));
}
}
return rsGroupInfoList;
@@ -383,7 +383,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
ByteArrayInputStream bis =
new ByteArrayInputStream(data, ProtobufUtil.lengthOfPBMagic(), data.length);
RSGroupInfoList
- .add(RSGroupProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis)));
+ .add(ProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis)));
}
}
LOG.debug("Read ZK GroupInfo count:" + RSGroupInfoList.size());
@@ -459,7 +459,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// populate puts
for (RSGroupInfo RSGroupInfo : groupMap.values()) {
- RSGroupProtos.RSGroupInfo proto = RSGroupProtobufUtil.toProtoGroupInfo(RSGroupInfo);
+ RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(RSGroupInfo);
Put p = new Put(Bytes.toBytes(RSGroupInfo.getName()));
p.addColumn(META_FAMILY_BYTES, META_QUALIFIER_BYTES, proto.toByteArray());
mutations.add(p);
@@ -532,7 +532,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
for (RSGroupInfo RSGroupInfo : newGroupMap.values()) {
String znode = ZNodePaths.joinZNode(groupBasePath, RSGroupInfo.getName());
- RSGroupProtos.RSGroupInfo proto = RSGroupProtobufUtil.toProtoGroupInfo(RSGroupInfo);
+ RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(RSGroupInfo);
LOG.debug("Updating znode: " + znode);
ZKUtil.createAndFailSilent(watcher, znode);
zkOps.add(ZKUtil.ZKUtilOp.deleteNodeFailSilent(znode));
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java
deleted file mode 100644
index 56e35e7..0000000
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupProtobufUtil.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hbase.rsgroup;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.net.Address;
-import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
-import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
-import org.apache.yetus.audience.InterfaceAudience;
-
-@InterfaceAudience.Private
-final class RSGroupProtobufUtil {
- private RSGroupProtobufUtil() {
- }
-
- static RSGroupInfo toGroupInfo(RSGroupProtos.RSGroupInfo proto) {
- RSGroupInfo RSGroupInfo = new RSGroupInfo(proto.getName());
- for(HBaseProtos.ServerName el: proto.getServersList()) {
- RSGroupInfo.addServer(Address.fromParts(el.getHostName(), el.getPort()));
- }
- for(HBaseProtos.TableName pTableName: proto.getTablesList()) {
- RSGroupInfo.addTable(ProtobufUtil.toTableName(pTableName));
- }
- return RSGroupInfo;
- }
-
- static RSGroupProtos.RSGroupInfo toProtoGroupInfo(RSGroupInfo pojo) {
- List<HBaseProtos.TableName> tables = new ArrayList<>(pojo.getTables().size());
- for(TableName arg: pojo.getTables()) {
- tables.add(ProtobufUtil.toProtoTableName(arg));
- }
- List<HBaseProtos.ServerName> hostports = new ArrayList<>(pojo.getServers().size());
- for(Address el: pojo.getServers()) {
- hostports.add(HBaseProtos.ServerName.newBuilder()
- .setHostName(el.getHostname())
- .setPort(el.getPort())
- .build());
- }
- return RSGroupProtos.RSGroupInfo.newBuilder().setName(pojo.getName())
- .addAllServers(hostports)
- .addAllTables(tables).build();
- }
-}
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
index 88a4339..2ad30e4 100644
--- a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
+++ b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
@@ -22,7 +22,6 @@ import java.io.IOException;
import java.util.List;
import java.util.Map;
import java.util.Set;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.ConnectionFactory;
@@ -129,7 +128,7 @@ public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
result.getValue(
RSGroupInfoManager.META_FAMILY_BYTES,
RSGroupInfoManager.META_QUALIFIER_BYTES));
- groupMap.put(proto.getName(), RSGroupProtobufUtil.toGroupInfo(proto));
+ groupMap.put(proto.getName(), ProtobufUtil.toGroupInfo(proto));
}
Assert.assertEquals(Sets.newHashSet(groupMap.values()),
Sets.newHashSet(wrapped.listRSGroups()));
@@ -141,7 +140,7 @@ public class VerifyingRSGroupAdminClient implements RSGroupAdmin {
ProtobufUtil.expectPBMagicPrefix(data);
ByteArrayInputStream bis = new ByteArrayInputStream(
data, ProtobufUtil.lengthOfPBMagic(), data.length);
- zList.add(RSGroupProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis)));
+ zList.add(ProtobufUtil.toGroupInfo(RSGroupProtos.RSGroupInfo.parseFrom(bis)));
}
}
Assert.assertEquals(zList.size(), groupMap.size());
[hbase] 03/14: HBASE-22676 Move all the code in hbase-rsgroup to
hbase-server and remove hbase-rsgroup module (#399)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 6aede3fbd339b9c855eb6029f0d17956989efd5a
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Tue Jul 23 09:40:02 2019 +0800
HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <op...@gmail.com>
---
hbase-assembly/src/main/assembly/components.xml | 8 -
.../src/main/assembly/hadoop-two-compat.xml | 1 -
hbase-it/pom.xml | 10 -
hbase-rsgroup/README.txt | 13 -
hbase-rsgroup/pom.xml | 262 ----------
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 559 ---------------------
hbase-rsgroup/src/test/resources/hbase-site.xml | 32 --
hbase-rsgroup/src/test/resources/log4j.properties | 68 ---
.../apache/hadoop/hbase/rsgroup/RSGroupAdmin.java | 0
.../hadoop/hbase/rsgroup/RSGroupAdminClient.java | 0
.../hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java | 226 +++++++++
.../hadoop/hbase/rsgroup/RSGroupAdminServer.java | 0
.../hbase/rsgroup/RSGroupAdminServiceImpl.java | 378 ++++++++++++++
.../hbase/rsgroup/RSGroupBasedLoadBalancer.java | 24 +-
.../hbase/rsgroup/RSGroupMajorCompactionTTL.java | 0
.../hadoop/hbase/rsgroup/RSGroupableBalancer.java | 0
.../balancer/RSGroupableBalancerTestBase.java | 0
.../balancer/TestRSGroupBasedLoadBalancer.java | 0
...lancerWithStochasticLoadBalancerAsInternal.java | 0
.../procedure/TestSCPWithReplicasWithRSGroup.java | 0
.../hadoop/hbase/rsgroup/TestEnableRSGroups.java | 0
.../rsgroup/TestRSGroupMajorCompactionTTL.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsBalance.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsBase.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsBasics.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsKillRS.java | 0
.../hbase/rsgroup/TestRSGroupsOfflineMode.java | 0
.../hadoop/hbase/rsgroup/TestRSGroupsWithACL.java | 77 ++-
.../hbase/rsgroup/VerifyingRSGroupAdminClient.java | 0
hbase-shell/pom.xml | 35 --
pom.xml | 24 -
33 files changed, 647 insertions(+), 1070 deletions(-)
diff --git a/hbase-assembly/src/main/assembly/components.xml b/hbase-assembly/src/main/assembly/components.xml
index 2eb16e7..4dd85ef 100644
--- a/hbase-assembly/src/main/assembly/components.xml
+++ b/hbase-assembly/src/main/assembly/components.xml
@@ -145,14 +145,6 @@
<fileMode>0644</fileMode>
</fileSet>
<fileSet>
- <directory>${project.basedir}/../hbase-rsgroup/target/</directory>
- <outputDirectory>lib</outputDirectory>
- <includes>
- <include>${rsgroup.test.jar}</include>
- </includes>
- <fileMode>0644</fileMode>
- </fileSet>
- <fileSet>
<directory>${project.basedir}/../hbase-mapreduce/target/</directory>
<outputDirectory>lib</outputDirectory>
<includes>
diff --git a/hbase-assembly/src/main/assembly/hadoop-two-compat.xml b/hbase-assembly/src/main/assembly/hadoop-two-compat.xml
index 91d3749..6a21b92 100644
--- a/hbase-assembly/src/main/assembly/hadoop-two-compat.xml
+++ b/hbase-assembly/src/main/assembly/hadoop-two-compat.xml
@@ -52,7 +52,6 @@
<include>org.apache.hbase:hbase-protocol-shaded</include>
<include>org.apache.hbase:hbase-replication</include>
<include>org.apache.hbase:hbase-rest</include>
- <include>org.apache.hbase:hbase-rsgroup</include>
<include>org.apache.hbase:hbase-server</include>
<include>org.apache.hbase:hbase-shell</include>
<include>org.apache.hbase:hbase-testing-util</include>
diff --git a/hbase-it/pom.xml b/hbase-it/pom.xml
index 0fe578a..6b173f0 100644
--- a/hbase-it/pom.xml
+++ b/hbase-it/pom.xml
@@ -183,16 +183,6 @@
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
- <artifactId>hbase-rsgroup</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-rsgroup</artifactId>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
</dependency>
<dependency>
diff --git a/hbase-rsgroup/README.txt b/hbase-rsgroup/README.txt
deleted file mode 100644
index b24aee6..0000000
--- a/hbase-rsgroup/README.txt
+++ /dev/null
@@ -1,13 +0,0 @@
-ON PROTOBUFS
-This maven module has protobuf definition files ('.protos') used by hbase
-Coprocessor Endpoints that ship with hbase core including tests. Coprocessor
-Endpoints are meant to be standalone, independent code not reliant on hbase
-internals. They define their Service using protobuf. The protobuf version
-they use can be distinct from that used by HBase internally since HBase started
-shading its protobuf references. Endpoints have no access to the shaded protobuf
-hbase uses. They do have access to the content of hbase-protocol -- the
-.protos found in here -- but avoid using as much of this as you can as it is
-liable to change.
-
-Generation of java files from protobuf .proto files included here is done as
-part of the build.
diff --git a/hbase-rsgroup/pom.xml b/hbase-rsgroup/pom.xml
deleted file mode 100644
index 1135939..0000000
--- a/hbase-rsgroup/pom.xml
+++ /dev/null
@@ -1,262 +0,0 @@
-<?xml version="1.0"?>
-<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
- <!--
- /**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
- -->
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <artifactId>hbase-build-configuration</artifactId>
- <groupId>org.apache.hbase</groupId>
- <version>3.0.0-SNAPSHOT</version>
- <relativePath>../hbase-build-configuration</relativePath>
- </parent>
- <artifactId>hbase-rsgroup</artifactId>
- <name>Apache HBase - RSGroup</name>
- <description>Regionserver Groups for HBase</description>
- <build>
- <plugins>
- <plugin>
- <!--Make it so assembly:single does nothing in here-->
- <artifactId>maven-assembly-plugin</artifactId>
- <configuration>
- <skipAssembly>true</skipAssembly>
- </configuration>
- </plugin>
- <!-- Make a jar and put the sources in the jar -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-source-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-checkstyle-plugin</artifactId>
- <configuration>
- <failOnViolation>true</failOnViolation>
- </configuration>
- </plugin>
- <plugin>
- <groupId>net.revelc.code</groupId>
- <artifactId>warbucks-maven-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
- <dependencies>
- <!-- Intra-project dependencies -->
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-annotations</artifactId>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-client</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-server</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-common</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-procedure</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-procedure</artifactId>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-protocol</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-protocol-shaded</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-testing-util</artifactId>
- <scope>test</scope>
- </dependency>
- <!-- General dependencies -->
- <dependency>
- <groupId>org.apache.commons</groupId>
- <artifactId>commons-lang3</artifactId>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase.thirdparty</groupId>
- <artifactId>hbase-shaded-miscellaneous</artifactId>
- </dependency>
- <dependency>
- <groupId>org.apache.zookeeper</groupId>
- <artifactId>zookeeper</artifactId>
- </dependency>
- <dependency>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.mockito</groupId>
- <artifactId>mockito-core</artifactId>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <scope>test</scope>
- </dependency>
- </dependencies>
- <profiles>
- <!-- Skip the tests in this module -->
- <profile>
- <id>skipRSGroupTests</id>
- <activation>
- <property>
- <name>skipRSGroupTests</name>
- </property>
- </activation>
- <properties>
- <surefire.skipFirstPart>true</surefire.skipFirstPart>
- <surefire.skipSecondPart>true</surefire.skipSecondPart>
- </properties>
- </profile>
- <!-- profile against Hadoop 2.x: This is the default. -->
- <profile>
- <id>hadoop-2.0</id>
- <activation>
- <property>
- <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
- <!--h2-->
- <name>!hadoop.profile</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>com.github.stephenc.findbugs</groupId>
- <artifactId>findbugs-annotations</artifactId>
- <optional>true</optional>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <exclusions>
- <exclusion>
- <groupId>net.java.dev.jets3t</groupId>
- <artifactId>jets3t</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet.jsp</groupId>
- <artifactId>jsp-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.mortbay.jetty</groupId>
- <artifactId>jetty</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-server</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-json</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>tomcat</groupId>
- <artifactId>jasper-compiler</artifactId>
- </exclusion>
- <exclusion>
- <groupId>tomcat</groupId>
- <artifactId>jasper-runtime</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.code.findbugs</groupId>
- <artifactId>jsr305</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- </dependencies>
- </profile>
- <!--
- profile for building against Hadoop 3.0.x. Activate using:
- mvn -Dhadoop.profile=3.0
- -->
- <profile>
- <id>hadoop-3.0</id>
- <activation>
- <property>
- <name>hadoop.profile</name>
- <value>3.0</value>
- </property>
- </activation>
- <properties>
- <hadoop.version>3.0-SNAPSHOT</hadoop.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </dependency>
- </dependencies>
- </profile>
- <profile>
- <id>eclipse-specific</id>
- <activation>
- <property>
- <name>m2e.version</name>
- </property>
- </activation>
- <build>
- <pluginManagement>
- <plugins>
- <!--This plugin's configuration is used to store Eclipse m2e settings
- only. It has no influence on the Maven build itself.-->
- <plugin>
- <groupId>org.eclipse.m2e</groupId>
- <artifactId>lifecycle-mapping</artifactId>
- <configuration>
- <lifecycleMappingMetadata>
- <pluginExecutions>
- </pluginExecutions>
- </lifecycleMappingMetadata>
- </configuration>
- </plugin>
- </plugins>
- </pluginManagement>
- </build>
- </profile>
- </profiles>
-</project>
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
deleted file mode 100644
index 8c6e010..0000000
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
+++ /dev/null
@@ -1,559 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hbase.rsgroup;
-
-import com.google.protobuf.RpcCallback;
-import com.google.protobuf.RpcController;
-import com.google.protobuf.Service;
-import java.io.IOException;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Optional;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.hadoop.hbase.CoprocessorEnvironment;
-import org.apache.hadoop.hbase.HBaseIOException;
-import org.apache.hadoop.hbase.HConstants;
-import org.apache.hadoop.hbase.MasterNotRunningException;
-import org.apache.hadoop.hbase.NamespaceDescriptor;
-import org.apache.hadoop.hbase.PleaseHoldException;
-import org.apache.hadoop.hbase.ServerName;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.RegionInfo;
-import org.apache.hadoop.hbase.client.SnapshotDescription;
-import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.constraint.ConstraintException;
-import org.apache.hadoop.hbase.coprocessor.CoreCoprocessor;
-import org.apache.hadoop.hbase.coprocessor.HasMasterServices;
-import org.apache.hadoop.hbase.coprocessor.MasterCoprocessor;
-import org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment;
-import org.apache.hadoop.hbase.coprocessor.MasterObserver;
-import org.apache.hadoop.hbase.coprocessor.ObserverContext;
-import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
-import org.apache.hadoop.hbase.ipc.RpcServer;
-import org.apache.hadoop.hbase.master.MasterServices;
-import org.apache.hadoop.hbase.net.Address;
-import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
-import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.AddRSGroupRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.AddRSGroupResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfServerRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfServerResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfTableRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfTableResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RSGroupAdminService;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupResponse;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
-import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersResponse;
-import org.apache.hadoop.hbase.security.User;
-import org.apache.hadoop.hbase.security.UserProvider;
-import org.apache.hadoop.hbase.security.access.AccessChecker;
-import org.apache.hadoop.hbase.security.access.Permission.Action;
-import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
-
-// TODO: Encapsulate MasterObserver functions into separate subclass.
-@CoreCoprocessor
-@InterfaceAudience.Private
-public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
- private static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminEndpoint.class);
-
- private MasterServices master = null;
- // Only instance of RSGroupInfoManager. RSGroup aware load balancers ask for this instance on
- // their setup.
- private RSGroupInfoManager groupInfoManager;
- private RSGroupAdminServer groupAdminServer;
- private final RSGroupAdminService groupAdminService = new RSGroupAdminServiceImpl();
- private AccessChecker accessChecker;
-
- /** Provider for mapping principal names to Users */
- private UserProvider userProvider;
-
- @Override
- public void start(CoprocessorEnvironment env) throws IOException {
- if (!(env instanceof HasMasterServices)) {
- throw new IOException("Does not implement HMasterServices");
- }
-
- master = ((HasMasterServices)env).getMasterServices();
- groupInfoManager = RSGroupInfoManagerImpl.getInstance(master);
- groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
- Class<?> clazz =
- master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
- if (!RSGroupableBalancer.class.isAssignableFrom(clazz)) {
- throw new IOException("Configured balancer does not support RegionServer groups.");
- }
- accessChecker = ((HasMasterServices) env).getMasterServices().getAccessChecker();
-
- // set the user-provider.
- this.userProvider = UserProvider.instantiate(env.getConfiguration());
- }
-
- @Override
- public void stop(CoprocessorEnvironment env) {
- }
-
- @Override
- public Iterable<Service> getServices() {
- return Collections.singleton(groupAdminService);
- }
-
- @Override
- public Optional<MasterObserver> getMasterObserver() {
- return Optional.of(this);
- }
-
- RSGroupInfoManager getGroupInfoManager() {
- return groupInfoManager;
- }
-
- /**
- * Implementation of RSGroupAdminService defined in RSGroupAdmin.proto.
- * This class calls {@link RSGroupAdminServer} for actual work, converts result to protocol
- * buffer response, handles exceptions if any occurred and then calls the {@code RpcCallback} with
- * the response.
- */
- private class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
- @Override
- public void getRSGroupInfo(RpcController controller,
- GetRSGroupInfoRequest request, RpcCallback<GetRSGroupInfoResponse> done) {
- GetRSGroupInfoResponse.Builder builder = GetRSGroupInfoResponse.newBuilder();
- String groupName = request.getRSGroupName();
- LOG.info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, group="
- + groupName);
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preGetRSGroupInfo(groupName);
- }
- checkPermission("getRSGroupInfo");
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
- }
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postGetRSGroupInfo(groupName);
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void getRSGroupInfoOfTable(RpcController controller,
- GetRSGroupInfoOfTableRequest request, RpcCallback<GetRSGroupInfoOfTableResponse> done) {
- GetRSGroupInfoOfTableResponse.Builder builder = GetRSGroupInfoOfTableResponse.newBuilder();
- TableName tableName = ProtobufUtil.toTableName(request.getTableName());
- LOG.info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, table="
- + tableName);
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preGetRSGroupInfoOfTable(tableName);
- }
- checkPermission("getRSGroupInfoOfTable");
- RSGroupInfo RSGroupInfo = groupAdminServer.getRSGroupInfoOfTable(tableName);
- if (RSGroupInfo != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
- }
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postGetRSGroupInfoOfTable(tableName);
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void moveServers(RpcController controller, MoveServersRequest request,
- RpcCallback<MoveServersResponse> done) {
- MoveServersResponse.Builder builder = MoveServersResponse.newBuilder();
- Set<Address> hostPorts = Sets.newHashSet();
- for (HBaseProtos.ServerName el : request.getServersList()) {
- hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
- }
- LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +" to rsgroup "
- + request.getTargetGroup());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preMoveServers(hostPorts, request.getTargetGroup());
- }
- checkPermission("moveServers");
- groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postMoveServers(hostPorts, request.getTargetGroup());
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void moveTables(RpcController controller, MoveTablesRequest request,
- RpcCallback<MoveTablesResponse> done) {
- MoveTablesResponse.Builder builder = MoveTablesResponse.newBuilder();
- Set<TableName> tables = new HashSet<>(request.getTableNameList().size());
- for (HBaseProtos.TableName tableName : request.getTableNameList()) {
- tables.add(ProtobufUtil.toTableName(tableName));
- }
- LOG.info(master.getClientIdAuditPrefix() + " move tables " + tables +" to rsgroup "
- + request.getTargetGroup());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preMoveTables(tables, request.getTargetGroup());
- }
- checkPermission("moveTables");
- groupAdminServer.moveTables(tables, request.getTargetGroup());
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postMoveTables(tables, request.getTargetGroup());
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void addRSGroup(RpcController controller, AddRSGroupRequest request,
- RpcCallback<AddRSGroupResponse> done) {
- AddRSGroupResponse.Builder builder = AddRSGroupResponse.newBuilder();
- LOG.info(master.getClientIdAuditPrefix() + " add rsgroup " + request.getRSGroupName());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
- }
- checkPermission("addRSGroup");
- groupAdminServer.addRSGroup(request.getRSGroupName());
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postAddRSGroup(request.getRSGroupName());
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void removeRSGroup(RpcController controller,
- RemoveRSGroupRequest request, RpcCallback<RemoveRSGroupResponse> done) {
- RemoveRSGroupResponse.Builder builder =
- RemoveRSGroupResponse.newBuilder();
- LOG.info(master.getClientIdAuditPrefix() + " remove rsgroup " + request.getRSGroupName());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preRemoveRSGroup(request.getRSGroupName());
- }
- checkPermission("removeRSGroup");
- groupAdminServer.removeRSGroup(request.getRSGroupName());
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postRemoveRSGroup(request.getRSGroupName());
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void balanceRSGroup(RpcController controller,
- BalanceRSGroupRequest request, RpcCallback<BalanceRSGroupResponse> done) {
- BalanceRSGroupResponse.Builder builder = BalanceRSGroupResponse.newBuilder();
- LOG.info(master.getClientIdAuditPrefix() + " balance rsgroup, group=" +
- request.getRSGroupName());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preBalanceRSGroup(request.getRSGroupName());
- }
- checkPermission("balanceRSGroup");
- boolean balancerRan = groupAdminServer.balanceRSGroup(request.getRSGroupName());
- builder.setBalanceRan(balancerRan);
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postBalanceRSGroup(request.getRSGroupName(),
- balancerRan);
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- builder.setBalanceRan(false);
- }
- done.run(builder.build());
- }
-
- @Override
- public void listRSGroupInfos(RpcController controller,
- ListRSGroupInfosRequest request, RpcCallback<ListRSGroupInfosResponse> done) {
- ListRSGroupInfosResponse.Builder builder = ListRSGroupInfosResponse.newBuilder();
- LOG.info(master.getClientIdAuditPrefix() + " list rsgroup");
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preListRSGroups();
- }
- checkPermission("listRSGroup");
- for (RSGroupInfo RSGroupInfo : groupAdminServer.listRSGroups()) {
- builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
- }
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postListRSGroups();
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void getRSGroupInfoOfServer(RpcController controller,
- GetRSGroupInfoOfServerRequest request, RpcCallback<GetRSGroupInfoOfServerResponse> done) {
- GetRSGroupInfoOfServerResponse.Builder builder = GetRSGroupInfoOfServerResponse.newBuilder();
- Address hp = Address.fromParts(request.getServer().getHostName(),
- request.getServer().getPort());
- LOG.info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, server="
- + hp);
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preGetRSGroupInfoOfServer(hp);
- }
- checkPermission("getRSGroupInfoOfServer");
- RSGroupInfo info = groupAdminServer.getRSGroupOfServer(hp);
- if (info != null) {
- builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(info));
- }
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postGetRSGroupInfoOfServer(hp);
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void moveServersAndTables(RpcController controller,
- MoveServersAndTablesRequest request, RpcCallback<MoveServersAndTablesResponse> done) {
- MoveServersAndTablesResponse.Builder builder = MoveServersAndTablesResponse.newBuilder();
- Set<Address> hostPorts = Sets.newHashSet();
- for (HBaseProtos.ServerName el : request.getServersList()) {
- hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
- }
- Set<TableName> tables = new HashSet<>(request.getTableNameList().size());
- for (HBaseProtos.TableName tableName : request.getTableNameList()) {
- tables.add(ProtobufUtil.toTableName(tableName));
- }
- LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts
- + " and tables " + tables + " to rsgroup" + request.getTargetGroup());
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preMoveServersAndTables(hostPorts, tables,
- request.getTargetGroup());
- }
- checkPermission("moveServersAndTables");
- groupAdminServer.moveServersAndTables(hostPorts, tables, request.getTargetGroup());
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postMoveServersAndTables(hostPorts, tables,
- request.getTargetGroup());
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
-
- @Override
- public void removeServers(RpcController controller,
- RemoveServersRequest request,
- RpcCallback<RemoveServersResponse> done) {
- RemoveServersResponse.Builder builder =
- RemoveServersResponse.newBuilder();
- Set<Address> servers = Sets.newHashSet();
- for (HBaseProtos.ServerName el : request.getServersList()) {
- servers.add(Address.fromParts(el.getHostName(), el.getPort()));
- }
- LOG.info(master.getClientIdAuditPrefix()
- + " remove decommissioned servers from rsgroup: " + servers);
- try {
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().preRemoveServers(servers);
- }
- checkPermission("removeServers");
- groupAdminServer.removeServers(servers);
- if (master.getMasterCoprocessorHost() != null) {
- master.getMasterCoprocessorHost().postRemoveServers(servers);
- }
- } catch (IOException e) {
- CoprocessorRpcUtils.setControllerException(controller, e);
- }
- done.run(builder.build());
- }
- }
-
- boolean rsgroupHasServersOnline(TableDescriptor desc) throws IOException {
- String groupName;
- try {
- groupName =
- master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
- .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (groupName == null) {
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
- } catch (MasterNotRunningException | PleaseHoldException e) {
- LOG.info("Master has not initialized yet; temporarily using default RSGroup '" +
- RSGroupInfo.DEFAULT_GROUP + "' for deploy of system table");
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
-
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo == null) {
- throw new ConstraintException(
- "Default RSGroup (" + groupName + ") for this table's " + "namespace does not exist.");
- }
-
- for (ServerName onlineServer : master.getServerManager().createDestinationServersList()) {
- if (rsGroupInfo.getServers().contains(onlineServer.getAddress())) {
- return true;
- }
- }
- return false;
- }
-
- void assignTableToGroup(TableDescriptor desc) throws IOException {
- String groupName =
- master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
- .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if (groupName == null) {
- groupName = RSGroupInfo.DEFAULT_GROUP;
- }
- RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
- if (rsGroupInfo == null) {
- throw new ConstraintException("Default RSGroup (" + groupName + ") for this table's "
- + "namespace does not exist.");
- }
- if (!rsGroupInfo.containsTable(desc.getTableName())) {
- LOG.debug("Pre-moving table " + desc.getTableName() + " to RSGroup " + groupName);
- groupAdminServer.moveTables(Sets.newHashSet(desc.getTableName()), groupName);
- }
- }
-
- /////////////////////////////////////////////////////////////////////////////
- // MasterObserver overrides
- /////////////////////////////////////////////////////////////////////////////
-
- @Override
- public void preCreateTableAction(
- final ObserverContext<MasterCoprocessorEnvironment> ctx,
- final TableDescriptor desc,
- final RegionInfo[] regions) throws IOException {
- if (!desc.getTableName().isSystemTable() && !rsgroupHasServersOnline(desc)) {
- throw new HBaseIOException("No online servers in the rsgroup, which table " +
- desc.getTableName().getNameAsString() + " belongs to");
- }
- }
-
- // Assign table to default RSGroup.
- @Override
- public void postCreateTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableDescriptor desc, RegionInfo[] regions) throws IOException {
- assignTableToGroup(desc);
- }
-
- // Remove table from its RSGroup.
- @Override
- public void postDeleteTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
- TableName tableName) throws IOException {
- try {
- RSGroupInfo group = groupAdminServer.getRSGroupInfoOfTable(tableName);
- if (group != null) {
- LOG.debug(String.format("Removing deleted table '%s' from rsgroup '%s'", tableName,
- group.getName()));
- groupAdminServer.moveTables(Sets.newHashSet(tableName), null);
- }
- } catch (IOException ex) {
- LOG.debug("Failed to perform RSGroup information cleanup for table: " + tableName, ex);
- }
- }
-
- @Override
- public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor ns) throws IOException {
- String group = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
- if(group != null && groupAdminServer.getRSGroupInfo(group) == null) {
- throw new ConstraintException("Region server group "+group+" does not exit");
- }
- }
-
- @Override
- public void preModifyNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
- NamespaceDescriptor currentNsDesc, NamespaceDescriptor newNsDesc) throws IOException {
- preCreateNamespace(ctx, newNsDesc);
- }
-
- @Override
- public void preCloneSnapshot(ObserverContext<MasterCoprocessorEnvironment> ctx,
- SnapshotDescription snapshot, TableDescriptor desc) throws IOException {
- assignTableToGroup(desc);
- }
-
- @Override
- public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
- List<ServerName> servers, List<ServerName> notClearedServers)
- throws IOException {
- Set<Address> clearedServer = servers.stream().
- filter(server -> !notClearedServers.contains(server)).
- map(ServerName::getAddress).
- collect(Collectors.toSet());
- if(!clearedServer.isEmpty()) {
- groupAdminServer.removeServers(clearedServer);
- }
- }
-
- public void checkPermission(String request) throws IOException {
- accessChecker.requirePermission(getActiveUser(), request, null, Action.ADMIN);
- }
-
- /**
- * Returns the active user to which authorization checks should be applied.
- * If we are in the context of an RPC call, the remote user is used,
- * otherwise the currently logged in user is used.
- */
- private User getActiveUser() throws IOException {
- // for non-rpc handling, fallback to system user
- Optional<User> optionalUser = RpcServer.getRequestUser();
- if (optionalUser.isPresent()) {
- return optionalUser.get();
- }
- return userProvider.getCurrent();
- }
-}
diff --git a/hbase-rsgroup/src/test/resources/hbase-site.xml b/hbase-rsgroup/src/test/resources/hbase-site.xml
deleted file mode 100644
index 99d2ab8..0000000
--- a/hbase-rsgroup/src/test/resources/hbase-site.xml
+++ /dev/null
@@ -1,32 +0,0 @@
-<?xml version="1.0"?>
-<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-<!--
-/**
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
--->
-<configuration>
- <property>
- <name>hbase.defaults.for.version.skip</name>
- <value>true</value>
- </property>
- <property>
- <name>hbase.hconnection.threads.keepalivetime</name>
- <value>3</value>
- </property>
-</configuration>
diff --git a/hbase-rsgroup/src/test/resources/log4j.properties b/hbase-rsgroup/src/test/resources/log4j.properties
deleted file mode 100644
index c322699..0000000
--- a/hbase-rsgroup/src/test/resources/log4j.properties
+++ /dev/null
@@ -1,68 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Define some default values that can be overridden by system properties
-hbase.root.logger=INFO,console
-hbase.log.dir=.
-hbase.log.file=hbase.log
-
-# Define the root logger to the system property "hbase.root.logger".
-log4j.rootLogger=${hbase.root.logger}
-
-# Logging Threshold
-log4j.threshold=ALL
-
-#
-# Daily Rolling File Appender
-#
-log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.DRFA.File=${hbase.log.dir}/${hbase.log.file}
-
-# Rollver at midnight
-log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
-
-# 30-day backup
-#log4j.appender.DRFA.MaxBackupIndex=30
-log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
-# Debugging Pattern format
-log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %C{2}(%L): %m%n
-
-
-#
-# console
-# Add "console" to rootlogger above if you want to use this
-#
-log4j.appender.console=org.apache.log4j.ConsoleAppender
-log4j.appender.console.target=System.err
-log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %C{2}(%L): %m%n
-
-# Custom Logging levels
-
-#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
-
-log4j.logger.org.apache.hadoop=WARN
-log4j.logger.org.apache.zookeeper=ERROR
-log4j.logger.org.apache.hadoop.hbase=DEBUG
-
-#These settings are workarounds against spurious logs from the minicluster.
-#See HBASE-4709
-log4j.logger.org.apache.hadoop.metrics2.impl.MetricsConfig=WARN
-log4j.logger.org.apache.hadoop.metrics2.impl.MetricsSinkAdapter=WARN
-log4j.logger.org.apache.hadoop.metrics2.impl.MetricsSystemImpl=WARN
-log4j.logger.org.apache.hadoop.metrics2.util.MBeans=WARN
-# Enable this to get detailed connection error/retry logging.
-# log4j.logger.org.apache.hadoop.hbase.client.ConnectionImplementation=TRACE
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
similarity index 100%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
similarity index 100%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminClient.java
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
new file mode 100644
index 0000000..2d5af04
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.rsgroup;
+
+import com.google.protobuf.Service;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hbase.CoprocessorEnvironment;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MasterNotRunningException;
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.PleaseHoldException;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.SnapshotDescription;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.constraint.ConstraintException;
+import org.apache.hadoop.hbase.coprocessor.CoreCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.HasMasterServices;
+import org.apache.hadoop.hbase.coprocessor.MasterCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.MasterCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.MasterObserver;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.net.Address;
+import org.apache.hadoop.hbase.security.UserProvider;
+import org.apache.hadoop.hbase.security.access.AccessChecker;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
+
+// TODO: Encapsulate MasterObserver functions into separate subclass.
+@CoreCoprocessor
+@InterfaceAudience.Private
+public class RSGroupAdminEndpoint implements MasterCoprocessor, MasterObserver {
+ static final Logger LOG = LoggerFactory.getLogger(RSGroupAdminEndpoint.class);
+
+ private MasterServices master;
+ // Only instance of RSGroupInfoManager. RSGroup aware load balancers ask for this instance on
+ // their setup.
+ private RSGroupInfoManager groupInfoManager;
+ private RSGroupAdminServer groupAdminServer;
+ private RSGroupAdminServiceImpl groupAdminService = new RSGroupAdminServiceImpl();
+
+ @Override
+ public void start(CoprocessorEnvironment env) throws IOException {
+ if (!(env instanceof HasMasterServices)) {
+ throw new IOException("Does not implement HMasterServices");
+ }
+
+ master = ((HasMasterServices) env).getMasterServices();
+ groupInfoManager = RSGroupInfoManagerImpl.getInstance(master);
+ groupAdminServer = new RSGroupAdminServer(master, groupInfoManager);
+ Class<?> clazz =
+ master.getConfiguration().getClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, null);
+ if (!RSGroupableBalancer.class.isAssignableFrom(clazz)) {
+ throw new IOException("Configured balancer does not support RegionServer groups.");
+ }
+ AccessChecker accessChecker = ((HasMasterServices) env).getMasterServices().getAccessChecker();
+
+ // set the user-provider.
+ UserProvider userProvider = UserProvider.instantiate(env.getConfiguration());
+ groupAdminService.initialize(master, groupAdminServer, accessChecker, userProvider);
+ }
+
+ @Override
+ public void stop(CoprocessorEnvironment env) {
+ }
+
+ @Override
+ public Iterable<Service> getServices() {
+ return Collections.singleton(groupAdminService);
+ }
+
+ @Override
+ public Optional<MasterObserver> getMasterObserver() {
+ return Optional.of(this);
+ }
+
+ RSGroupInfoManager getGroupInfoManager() {
+ return groupInfoManager;
+ }
+
+ @VisibleForTesting
+ RSGroupAdminServiceImpl getGroupAdminService() {
+ return groupAdminService;
+ }
+
+ private void assignTableToGroup(TableDescriptor desc) throws IOException {
+ String groupName =
+ master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
+ .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (groupName == null) {
+ groupName = RSGroupInfo.DEFAULT_GROUP;
+ }
+ RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
+ if (rsGroupInfo == null) {
+ throw new ConstraintException(
+ "Default RSGroup (" + groupName + ") for this table's namespace does not exist.");
+ }
+ if (!rsGroupInfo.containsTable(desc.getTableName())) {
+ LOG.debug("Pre-moving table " + desc.getTableName() + " to RSGroup " + groupName);
+ groupAdminServer.moveTables(Sets.newHashSet(desc.getTableName()), groupName);
+ }
+ }
+
+ /////////////////////////////////////////////////////////////////////////////
+ // MasterObserver overrides
+ /////////////////////////////////////////////////////////////////////////////
+
+ private boolean rsgroupHasServersOnline(TableDescriptor desc) throws IOException {
+ String groupName;
+ try {
+ groupName = master.getClusterSchema().getNamespace(desc.getTableName().getNamespaceAsString())
+ .getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (groupName == null) {
+ groupName = RSGroupInfo.DEFAULT_GROUP;
+ }
+ } catch (MasterNotRunningException | PleaseHoldException e) {
+ LOG.info("Master has not initialized yet; temporarily using default RSGroup '" +
+ RSGroupInfo.DEFAULT_GROUP + "' for deploy of system table");
+ groupName = RSGroupInfo.DEFAULT_GROUP;
+ }
+
+ RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
+ if (rsGroupInfo == null) {
+ throw new ConstraintException(
+ "Default RSGroup (" + groupName + ") for this table's " + "namespace does not exist.");
+ }
+
+ for (ServerName onlineServer : master.getServerManager().createDestinationServersList()) {
+ if (rsGroupInfo.getServers().contains(onlineServer.getAddress())) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public void preCreateTableAction(final ObserverContext<MasterCoprocessorEnvironment> ctx,
+ final TableDescriptor desc, final RegionInfo[] regions) throws IOException {
+ if (!desc.getTableName().isSystemTable() && !rsgroupHasServersOnline(desc)) {
+ throw new HBaseIOException("No online servers in the rsgroup, which table " +
+ desc.getTableName().getNameAsString() + " belongs to");
+ }
+ }
+
+ // Assign table to default RSGroup.
+ @Override
+ public void postCreateTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ TableDescriptor desc, RegionInfo[] regions) throws IOException {
+ assignTableToGroup(desc);
+ }
+
+ // Remove table from its RSGroup.
+ @Override
+ public void postDeleteTable(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ TableName tableName) throws IOException {
+ try {
+ RSGroupInfo group = groupAdminServer.getRSGroupInfoOfTable(tableName);
+ if (group != null) {
+ LOG.debug(String.format("Removing deleted table '%s' from rsgroup '%s'", tableName,
+ group.getName()));
+ groupAdminServer.moveTables(Sets.newHashSet(tableName), null);
+ }
+ } catch (IOException ex) {
+ LOG.debug("Failed to perform RSGroup information cleanup for table: " + tableName, ex);
+ }
+ }
+
+ @Override
+ public void preCreateNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ NamespaceDescriptor ns) throws IOException {
+ String group = ns.getConfigurationValue(RSGroupInfo.NAMESPACE_DESC_PROP_GROUP);
+ if (group != null && groupAdminServer.getRSGroupInfo(group) == null) {
+ throw new ConstraintException("Region server group " + group + " does not exit");
+ }
+ }
+
+ @Override
+ public void preModifyNamespace(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ NamespaceDescriptor currentNsDesc, NamespaceDescriptor newNsDesc) throws IOException {
+ preCreateNamespace(ctx, newNsDesc);
+ }
+
+ @Override
+ public void preCloneSnapshot(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ SnapshotDescription snapshot, TableDescriptor desc) throws IOException {
+ assignTableToGroup(desc);
+ }
+
+ @Override
+ public void postClearDeadServers(ObserverContext<MasterCoprocessorEnvironment> ctx,
+ List<ServerName> servers, List<ServerName> notClearedServers) throws IOException {
+ Set<Address> clearedServer =
+ servers.stream().filter(server -> !notClearedServers.contains(server))
+ .map(ServerName::getAddress).collect(Collectors.toSet());
+ if (!clearedServer.isEmpty()) {
+ groupAdminServer.removeServers(clearedServer);
+ }
+ }
+}
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
similarity index 100%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
new file mode 100644
index 0000000..918a4fe
--- /dev/null
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
@@ -0,0 +1,378 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.rsgroup;
+
+import com.google.protobuf.RpcCallback;
+import com.google.protobuf.RpcController;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils;
+import org.apache.hadoop.hbase.ipc.RpcServer;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.net.Address;
+import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.AddRSGroupRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.AddRSGroupResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.BalanceRSGroupResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfServerRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfServerResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfTableRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoOfTableResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.GetRSGroupInfoResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.ListRSGroupInfosResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersAndTablesResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveServersResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.MoveTablesResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveRSGroupResponse;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersRequest;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos.RemoveServersResponse;
+import org.apache.hadoop.hbase.security.User;
+import org.apache.hadoop.hbase.security.UserProvider;
+import org.apache.hadoop.hbase.security.access.AccessChecker;
+import org.apache.hadoop.hbase.security.access.Permission.Action;
+
+import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
+
+/**
+ * Implementation of RSGroupAdminService defined in RSGroupAdmin.proto. This class calls
+ * {@link RSGroupAdminServer} for actual work, converts result to protocol buffer response, handles
+ * exceptions if any occurred and then calls the {@code RpcCallback} with the response.
+ */
+class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
+
+ private MasterServices master;
+
+ private RSGroupAdminServer groupAdminServer;
+
+ private AccessChecker accessChecker;
+
+ /** Provider for mapping principal names to Users */
+ private UserProvider userProvider;
+
+ RSGroupAdminServiceImpl() {
+ }
+
+ void initialize(MasterServices master, RSGroupAdminServer groupAdminServer,
+ AccessChecker accessChecker, UserProvider userProvider) {
+ this.master = master;
+ this.groupAdminServer = groupAdminServer;
+ this.accessChecker = accessChecker;
+ this.userProvider = userProvider;
+ }
+
+ @VisibleForTesting
+ void checkPermission(String request) throws IOException {
+ accessChecker.requirePermission(getActiveUser(), request, null, Action.ADMIN);
+ }
+
+ /**
+ * Returns the active user to which authorization checks should be applied. If we are in the
+ * context of an RPC call, the remote user is used, otherwise the currently logged in user is
+ * used.
+ */
+ private User getActiveUser() throws IOException {
+ // for non-rpc handling, fallback to system user
+ Optional<User> optionalUser = RpcServer.getRequestUser();
+ if (optionalUser.isPresent()) {
+ return optionalUser.get();
+ }
+ return userProvider.getCurrent();
+ }
+
+ @Override
+ public void getRSGroupInfo(RpcController controller, GetRSGroupInfoRequest request,
+ RpcCallback<GetRSGroupInfoResponse> done) {
+ GetRSGroupInfoResponse.Builder builder = GetRSGroupInfoResponse.newBuilder();
+ String groupName = request.getRSGroupName();
+ RSGroupAdminEndpoint.LOG.info(
+ master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, group=" + groupName);
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preGetRSGroupInfo(groupName);
+ }
+ checkPermission("getRSGroupInfo");
+ RSGroupInfo rsGroupInfo = groupAdminServer.getRSGroupInfo(groupName);
+ if (rsGroupInfo != null) {
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(rsGroupInfo));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postGetRSGroupInfo(groupName);
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void getRSGroupInfoOfTable(RpcController controller, GetRSGroupInfoOfTableRequest request,
+ RpcCallback<GetRSGroupInfoOfTableResponse> done) {
+ GetRSGroupInfoOfTableResponse.Builder builder = GetRSGroupInfoOfTableResponse.newBuilder();
+ TableName tableName = ProtobufUtil.toTableName(request.getTableName());
+ RSGroupAdminEndpoint.LOG.info(
+ master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, table=" + tableName);
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preGetRSGroupInfoOfTable(tableName);
+ }
+ checkPermission("getRSGroupInfoOfTable");
+ RSGroupInfo RSGroupInfo = groupAdminServer.getRSGroupInfoOfTable(tableName);
+ if (RSGroupInfo != null) {
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postGetRSGroupInfoOfTable(tableName);
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void moveServers(RpcController controller, MoveServersRequest request,
+ RpcCallback<MoveServersResponse> done) {
+ MoveServersResponse.Builder builder = MoveServersResponse.newBuilder();
+ Set<Address> hostPorts = Sets.newHashSet();
+ for (HBaseProtos.ServerName el : request.getServersList()) {
+ hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +
+ " to rsgroup " + request.getTargetGroup());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preMoveServers(hostPorts, request.getTargetGroup());
+ }
+ checkPermission("moveServers");
+ groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postMoveServers(hostPorts, request.getTargetGroup());
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void moveTables(RpcController controller, MoveTablesRequest request,
+ RpcCallback<MoveTablesResponse> done) {
+ MoveTablesResponse.Builder builder = MoveTablesResponse.newBuilder();
+ Set<TableName> tables = new HashSet<>(request.getTableNameList().size());
+ for (HBaseProtos.TableName tableName : request.getTableNameList()) {
+ tables.add(ProtobufUtil.toTableName(tableName));
+ }
+ RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move tables " + tables +
+ " to rsgroup " + request.getTargetGroup());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preMoveTables(tables, request.getTargetGroup());
+ }
+ checkPermission("moveTables");
+ groupAdminServer.moveTables(tables, request.getTargetGroup());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postMoveTables(tables, request.getTargetGroup());
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void addRSGroup(RpcController controller, AddRSGroupRequest request,
+ RpcCallback<AddRSGroupResponse> done) {
+ AddRSGroupResponse.Builder builder = AddRSGroupResponse.newBuilder();
+ RSGroupAdminEndpoint.LOG
+ .info(master.getClientIdAuditPrefix() + " add rsgroup " + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preAddRSGroup(request.getRSGroupName());
+ }
+ checkPermission("addRSGroup");
+ groupAdminServer.addRSGroup(request.getRSGroupName());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postAddRSGroup(request.getRSGroupName());
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void removeRSGroup(RpcController controller, RemoveRSGroupRequest request,
+ RpcCallback<RemoveRSGroupResponse> done) {
+ RemoveRSGroupResponse.Builder builder = RemoveRSGroupResponse.newBuilder();
+ RSGroupAdminEndpoint.LOG
+ .info(master.getClientIdAuditPrefix() + " remove rsgroup " + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preRemoveRSGroup(request.getRSGroupName());
+ }
+ checkPermission("removeRSGroup");
+ groupAdminServer.removeRSGroup(request.getRSGroupName());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postRemoveRSGroup(request.getRSGroupName());
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void balanceRSGroup(RpcController controller, BalanceRSGroupRequest request,
+ RpcCallback<BalanceRSGroupResponse> done) {
+ BalanceRSGroupResponse.Builder builder = BalanceRSGroupResponse.newBuilder();
+ RSGroupAdminEndpoint.LOG.info(
+ master.getClientIdAuditPrefix() + " balance rsgroup, group=" + request.getRSGroupName());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preBalanceRSGroup(request.getRSGroupName());
+ }
+ checkPermission("balanceRSGroup");
+ boolean balancerRan = groupAdminServer.balanceRSGroup(request.getRSGroupName());
+ builder.setBalanceRan(balancerRan);
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postBalanceRSGroup(request.getRSGroupName(), balancerRan);
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ builder.setBalanceRan(false);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void listRSGroupInfos(RpcController controller, ListRSGroupInfosRequest request,
+ RpcCallback<ListRSGroupInfosResponse> done) {
+ ListRSGroupInfosResponse.Builder builder = ListRSGroupInfosResponse.newBuilder();
+ RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " list rsgroup");
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preListRSGroups();
+ }
+ checkPermission("listRSGroup");
+ for (RSGroupInfo RSGroupInfo : groupAdminServer.listRSGroups()) {
+ builder.addRSGroupInfo(ProtobufUtil.toProtoGroupInfo(RSGroupInfo));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postListRSGroups();
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void getRSGroupInfoOfServer(RpcController controller,
+ GetRSGroupInfoOfServerRequest request, RpcCallback<GetRSGroupInfoOfServerResponse> done) {
+ GetRSGroupInfoOfServerResponse.Builder builder = GetRSGroupInfoOfServerResponse.newBuilder();
+ Address hp =
+ Address.fromParts(request.getServer().getHostName(), request.getServer().getPort());
+ RSGroupAdminEndpoint.LOG
+ .info(master.getClientIdAuditPrefix() + " initiates rsgroup info retrieval, server=" + hp);
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preGetRSGroupInfoOfServer(hp);
+ }
+ checkPermission("getRSGroupInfoOfServer");
+ RSGroupInfo info = groupAdminServer.getRSGroupOfServer(hp);
+ if (info != null) {
+ builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(info));
+ }
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postGetRSGroupInfoOfServer(hp);
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void moveServersAndTables(RpcController controller, MoveServersAndTablesRequest request,
+ RpcCallback<MoveServersAndTablesResponse> done) {
+ MoveServersAndTablesResponse.Builder builder = MoveServersAndTablesResponse.newBuilder();
+ Set<Address> hostPorts = Sets.newHashSet();
+ for (HBaseProtos.ServerName el : request.getServersList()) {
+ hostPorts.add(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ Set<TableName> tables = new HashSet<>(request.getTableNameList().size());
+ for (HBaseProtos.TableName tableName : request.getTableNameList()) {
+ tables.add(ProtobufUtil.toTableName(tableName));
+ }
+ RSGroupAdminEndpoint.LOG.info(master.getClientIdAuditPrefix() + " move servers " + hostPorts +
+ " and tables " + tables + " to rsgroup" + request.getTargetGroup());
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preMoveServersAndTables(hostPorts, tables,
+ request.getTargetGroup());
+ }
+ checkPermission("moveServersAndTables");
+ groupAdminServer.moveServersAndTables(hostPorts, tables, request.getTargetGroup());
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postMoveServersAndTables(hostPorts, tables,
+ request.getTargetGroup());
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+
+ @Override
+ public void removeServers(RpcController controller, RemoveServersRequest request,
+ RpcCallback<RemoveServersResponse> done) {
+ RemoveServersResponse.Builder builder = RemoveServersResponse.newBuilder();
+ Set<Address> servers = Sets.newHashSet();
+ for (HBaseProtos.ServerName el : request.getServersList()) {
+ servers.add(Address.fromParts(el.getHostName(), el.getPort()));
+ }
+ RSGroupAdminEndpoint.LOG.info(
+ master.getClientIdAuditPrefix() + " remove decommissioned servers from rsgroup: " + servers);
+ try {
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().preRemoveServers(servers);
+ }
+ checkPermission("removeServers");
+ groupAdminServer.removeServers(servers);
+ if (master.getMasterCoprocessorHost() != null) {
+ master.getMasterCoprocessorHost().postRemoveServers(servers);
+ }
+ } catch (IOException e) {
+ CoprocessorRpcUtils.setControllerException(controller, e);
+ }
+ done.run(builder.build());
+ }
+}
\ No newline at end of file
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
similarity index 97%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index 7394f55..be76805 100644
--- a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -15,7 +15,6 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
@@ -56,18 +55,17 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
/**
- * GroupBasedLoadBalancer, used when Region Server Grouping is configured (HBase-6721)
- * It does region balance based on a table's group membership.
- *
- * Most assignment methods contain two exclusive code paths: Online - when the group
- * table is online and Offline - when it is unavailable.
- *
- * During Offline, assignments are assigned based on cached information in zookeeper.
- * If unavailable (ie bootstrap) then regions are assigned randomly.
- *
- * Once the GROUP table has been assigned, the balancer switches to Online and will then
- * start providing appropriate assignments for user tables.
- *
+ * GroupBasedLoadBalancer, used when Region Server Grouping is configured (HBase-6721) It does
+ * region balance based on a table's group membership.
+ * <p/>
+ * Most assignment methods contain two exclusive code paths: Online - when the group table is online
+ * and Offline - when it is unavailable.
+ * <p/>
+ * During Offline, assignments are assigned based on cached information in zookeeper. If unavailable
+ * (ie bootstrap) then regions are assigned randomly.
+ * <p/>
+ * Once the GROUP table has been assigned, the balancer switches to Online and will then start
+ * providing appropriate assignments for user tables.
*/
@InterfaceAudience.Private
public class RSGroupBasedLoadBalancer implements RSGroupableBalancer {
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
similarity index 100%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupMajorCompactionTTL.java
diff --git a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
similarity index 100%
rename from hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
rename to hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupableBalancer.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/RSGroupableBalancerTestBase.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancerWithStochasticLoadBalancerAsInternal.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSCPWithReplicasWithRSGroup.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSCPWithReplicasWithRSGroup.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSCPWithReplicasWithRSGroup.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSCPWithReplicasWithRSGroup.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroups.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroups.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroups.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroups.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBalance.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBasics.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsKillRS.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
similarity index 81%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
index 0278e3c..c1c157a 100644
--- a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsWithACL.java
@@ -39,7 +39,6 @@ import org.apache.hadoop.hbase.security.access.SecureTestUtil;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.testclassification.SecurityTests;
import org.apache.hadoop.hbase.util.Bytes;
-
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
@@ -49,11 +48,11 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Performs authorization checks for rsgroup operations, according to different
- * levels of authorized users.
+ * Performs authorization checks for rsgroup operations, according to different levels of authorized
+ * users.
*/
-@Category({SecurityTests.class, MediumTests.class})
-public class TestRSGroupsWithACL extends SecureTestUtil{
+@Category({ SecurityTests.class, MediumTests.class })
+public class TestRSGroupsWithACL extends SecureTestUtil {
@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
@@ -98,8 +97,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
public static void setupBeforeClass() throws Exception {
// setup configuration
conf = TEST_UTIL.getConfiguration();
- conf.set(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
- RSGroupBasedLoadBalancer.class.getName());
+ conf.set(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, RSGroupBasedLoadBalancer.class.getName());
// Enable security
enableSecurity(conf);
// Verify enableSecurity sets up what we require
@@ -108,8 +106,8 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
configureRSGroupAdminEndpoint(conf);
TEST_UTIL.startMiniCluster();
- rsGroupAdminEndpoint = (RSGroupAdminEndpoint) TEST_UTIL.getMiniHBaseCluster().getMaster().
- getMasterCoprocessorHost().findCoprocessor(RSGroupAdminEndpoint.class.getName());
+ rsGroupAdminEndpoint = (RSGroupAdminEndpoint) TEST_UTIL.getMiniHBaseCluster().getMaster()
+ .getMasterCoprocessorHost().findCoprocessor(RSGroupAdminEndpoint.class.getName());
// Wait for the ACL table to become available
TEST_UTIL.waitUntilAllRegionsAssigned(PermissionStorage.ACL_TABLE_NAME);
@@ -141,31 +139,21 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
cfd.setMaxVersions(100);
tableBuilder.setColumnFamily(cfd.build());
tableBuilder.setValue(TableDescriptorBuilder.OWNER, USER_OWNER.getShortName());
- createTable(TEST_UTIL, tableBuilder.build(),
- new byte[][] { Bytes.toBytes("s") });
+ createTable(TEST_UTIL, tableBuilder.build(), new byte[][] { Bytes.toBytes("s") });
// Set up initial grants
- grantGlobal(TEST_UTIL, USER_ADMIN.getShortName(),
- Permission.Action.ADMIN,
- Permission.Action.CREATE,
- Permission.Action.READ,
- Permission.Action.WRITE);
+ grantGlobal(TEST_UTIL, USER_ADMIN.getShortName(), Permission.Action.ADMIN,
+ Permission.Action.CREATE, Permission.Action.READ, Permission.Action.WRITE);
- grantOnTable(TEST_UTIL, USER_RW.getShortName(),
- TEST_TABLE, TEST_FAMILY, null,
- Permission.Action.READ,
- Permission.Action.WRITE);
+ grantOnTable(TEST_UTIL, USER_RW.getShortName(), TEST_TABLE, TEST_FAMILY, null,
+ Permission.Action.READ, Permission.Action.WRITE);
// USER_CREATE is USER_RW plus CREATE permissions
- grantOnTable(TEST_UTIL, USER_CREATE.getShortName(),
- TEST_TABLE, null, null,
- Permission.Action.CREATE,
- Permission.Action.READ,
- Permission.Action.WRITE);
+ grantOnTable(TEST_UTIL, USER_CREATE.getShortName(), TEST_TABLE, null, null,
+ Permission.Action.CREATE, Permission.Action.READ, Permission.Action.WRITE);
- grantOnTable(TEST_UTIL, USER_RO.getShortName(),
- TEST_TABLE, TEST_FAMILY, null,
- Permission.Action.READ);
+ grantOnTable(TEST_UTIL, USER_RO.getShortName(), TEST_TABLE, TEST_FAMILY, null,
+ Permission.Action.READ);
grantGlobal(TEST_UTIL, toGroupEntry(GROUP_ADMIN), Permission.Action.ADMIN);
grantGlobal(TEST_UTIL, toGroupEntry(GROUP_CREATE), Permission.Action.CREATE);
@@ -174,8 +162,8 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
assertEquals(4, PermissionStorage.getTablePermissions(conf, TEST_TABLE).size());
try {
- assertEquals(4, AccessControlClient.getUserPermissions(systemUserConnection,
- TEST_TABLE.toString()).size());
+ assertEquals(4,
+ AccessControlClient.getUserPermissions(systemUserConnection, TEST_TABLE.toString()).size());
} catch (AssertionError e) {
fail(e.getMessage());
} catch (Throwable e) {
@@ -210,14 +198,13 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
coprocessors += "," + currentCoprocessors;
}
conf.set(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY, coprocessors);
- conf.set(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
- RSGroupBasedLoadBalancer.class.getName());
+ conf.set(HConstants.HBASE_MASTER_LOADBALANCER_CLASS, RSGroupBasedLoadBalancer.class.getName());
}
@Test
public void testGetRSGroupInfo() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("getRSGroupInfo");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfo");
return null;
};
@@ -227,7 +214,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testGetRSGroupInfoOfTable() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("getRSGroupInfoOfTable");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfoOfTable");
return null;
};
@@ -237,7 +224,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testMoveServers() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("moveServers");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveServers");
return null;
};
@@ -247,7 +234,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testMoveTables() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("moveTables");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveTables");
return null;
};
@@ -257,7 +244,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testAddRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("addRSGroup");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("addRSGroup");
return null;
};
@@ -267,7 +254,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testRemoveRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("removeRSGroup");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("removeRSGroup");
return null;
};
@@ -277,7 +264,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testBalanceRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("balanceRSGroup");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("balanceRSGroup");
return null;
};
@@ -287,7 +274,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testListRSGroup() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("listRSGroup");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("listRSGroup");
return null;
};
@@ -297,7 +284,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testGetRSGroupInfoOfServer() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("getRSGroupInfoOfServer");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("getRSGroupInfoOfServer");
return null;
};
@@ -307,7 +294,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testMoveServersAndTables() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("moveServersAndTables");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("moveServersAndTables");
return null;
};
@@ -317,7 +304,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
@Test
public void testRemoveServers() throws Exception {
AccessTestAction action = () -> {
- rsGroupAdminEndpoint.checkPermission("removeServers");
+ rsGroupAdminEndpoint.getGroupAdminService().checkPermission("removeServers");
return null;
};
@@ -326,7 +313,7 @@ public class TestRSGroupsWithACL extends SecureTestUtil{
private void validateAdminPermissions(AccessTestAction action) throws Exception {
verifyAllowed(action, SUPERUSER, USER_ADMIN, USER_GROUP_ADMIN);
- verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO,
- USER_NONE, USER_GROUP_READ, USER_GROUP_WRITE, USER_GROUP_CREATE);
+ verifyDenied(action, USER_CREATE, USER_OWNER, USER_RW, USER_RO, USER_NONE, USER_GROUP_READ,
+ USER_GROUP_WRITE, USER_GROUP_CREATE);
}
}
diff --git a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
similarity index 100%
rename from hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
rename to hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdminClient.java
diff --git a/hbase-shell/pom.xml b/hbase-shell/pom.xml
index 7483da5..a6b62ad 100644
--- a/hbase-shell/pom.xml
+++ b/hbase-shell/pom.xml
@@ -187,41 +187,6 @@
</dependency>
</dependencies>
<profiles>
- <profile>
- <id>rsgroup</id>
- <activation>
- <property>
- <name>!skip-rsgroup</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-rsgroup</artifactId>
- </dependency>
- </dependencies>
- <build>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>build-helper-maven-plugin</artifactId>
- <executions>
- <execution>
- <id>add-test-source</id>
- <goals>
- <goal>add-test-source</goal>
- </goals>
- <configuration>
- <sources>
- <source>src/test/rsgroup</source>
- </sources>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
<!-- Skip the tests in this module -->
<profile>
<id>skipShellTests</id>
diff --git a/pom.xml b/pom.xml
index 4f680c1..923dfc9 100755
--- a/pom.xml
+++ b/pom.xml
@@ -1540,7 +1540,6 @@
<procedure.test.jar>hbase-procedure-${project.version}-tests.jar</procedure.test.jar>
<it.test.jar>hbase-it-${project.version}-tests.jar</it.test.jar>
<annotations.test.jar>hbase-annotations-${project.version}-tests.jar</annotations.test.jar>
- <rsgroup.test.jar>hbase-rsgroup-${project.version}-tests.jar</rsgroup.test.jar>
<mapreduce.test.jar>hbase-mapreduce-${project.version}-tests.jar</mapreduce.test.jar>
<zookeeper.test.jar>hbase-zookeeper-${project.version}-tests.jar</zookeeper.test.jar>
<shell-executable>bash</shell-executable>
@@ -1671,18 +1670,6 @@
<scope>test</scope>
</dependency>
<dependency>
- <artifactId>hbase-rsgroup</artifactId>
- <groupId>org.apache.hbase</groupId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <artifactId>hbase-rsgroup</artifactId>
- <groupId>org.apache.hbase</groupId>
- <version>${project.version}</version>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
<artifactId>hbase-replication</artifactId>
<groupId>org.apache.hbase</groupId>
<version>${project.version}</version>
@@ -2321,17 +2308,6 @@
-->
<profiles>
<profile>
- <id>rsgroup</id>
- <activation>
- <property>
- <name>!skip-rsgroup</name>
- </property>
- </activation>
- <modules>
- <module>hbase-rsgroup</module>
- </modules>
- </profile>
- <profile>
<id>build-with-jdk8</id>
<activation>
<jdk>[1.8,)</jdk>
[hbase] 07/14: HBASE-22819 Automatically migrate the rs group
config for table after HBASE-22695 (#498)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 3dc1b280948598799b29f8a2fcdd1f4af4b7c3ba
Author: Duo Zhang <zh...@apache.org>
AuthorDate: Sun Aug 25 11:19:04 2019 +0800
HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zg...@apache.org>
---
.../apache/hadoop/hbase/rsgroup/RSGroupInfo.java | 4 +-
.../hadoop/hbase/rsgroup/RSGroupAdminServer.java | 2 +-
.../hbase/rsgroup/RSGroupAdminServiceImpl.java | 2 +-
.../hadoop/hbase/rsgroup/RSGroupInfoManager.java | 14 +-
.../hbase/rsgroup/RSGroupInfoManagerImpl.java | 158 +++++++++++++++---
.../apache/hadoop/hbase/rsgroup/RSGroupUtil.java | 39 ++---
.../hbase/rsgroup/TestMigrateRSGroupInfo.java | 179 +++++++++++++++++++++
.../hadoop/hbase/rsgroup/TestRSGroupsBase.java | 3 +-
8 files changed, 343 insertions(+), 58 deletions(-)
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
index ad55d1f..817e237 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
@@ -19,10 +19,8 @@
package org.apache.hadoop.hbase.rsgroup;
import java.util.Collection;
-import java.util.Set;
import java.util.SortedSet;
import java.util.TreeSet;
-
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -104,7 +102,7 @@ public class RSGroupInfo {
/**
* Get list of servers.
*/
- public Set<Address> getServers() {
+ public SortedSet<Address> getServers() {
return servers;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
index 59950e1..1e324e5 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
@@ -61,7 +61,7 @@ public class RSGroupAdminServer implements RSGroupAdmin {
"one server in 'default' RSGroup.";
private MasterServices master;
- private final RSGroupInfoManager rsGroupInfoManager;
+ final RSGroupInfoManager rsGroupInfoManager;
/** Define the config key of retries threshold when movements failed */
//made package private for testing
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
index 6bc4519..749d353 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServiceImpl.java
@@ -164,7 +164,7 @@ class RSGroupAdminServiceImpl extends RSGroupAdminProtos.RSGroupAdminService {
}
checkPermission("getRSGroupInfoOfTable");
Optional<RSGroupInfo> optGroup =
- RSGroupUtil.getRSGroupInfo(master, groupAdminServer, tableName);
+ RSGroupUtil.getRSGroupInfo(master, groupAdminServer.rsGroupInfoManager, tableName);
if (optGroup.isPresent()) {
builder.setRSGroupInfo(ProtobufUtil.toProtoGroupInfo(fillTables(optGroup.get())));
} else {
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
index 28f7c1f..1b9f3ef 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hbase.rsgroup;
import java.io.IOException;
import java.util.List;
import java.util.Set;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.net.Address;
import org.apache.yetus.audience.InterfaceAudience;
@@ -68,11 +69,6 @@ public interface RSGroupInfoManager {
List<RSGroupInfo> listRSGroups() throws IOException;
/**
- * Refresh/reload the group information from the persistent store
- */
- void refresh() throws IOException;
-
- /**
* Whether the manager is able to fully return group metadata
* @return whether the manager is in online mode
*/
@@ -83,4 +79,12 @@ public interface RSGroupInfoManager {
* @param servers set of servers to remove
*/
void removeServers(Set<Address> servers) throws IOException;
+
+ /**
+ * Get {@code RSGroupInfo} for the given table.
+ * @deprecated Since 3.0.0, will be removed in 4.0.0. Only for compatibility, where we upgrade
+ * from a version that stores table names for a rs group in the {@code RSGroupInfo}.
+ */
+ @Deprecated
+ RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException;
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
index eaf23f3..6725066 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hbase.rsgroup;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
@@ -33,6 +34,7 @@ import org.apache.hadoop.hbase.Coprocessor;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.NamespaceDescriptor;
import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableDescriptors;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.AsyncClusterConnection;
import org.apache.hadoop.hbase.client.AsyncTable;
@@ -76,6 +78,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableMap;
import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.hbase.thirdparty.com.google.common.collect.Maps;
import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
@@ -104,9 +107,6 @@ import org.apache.hbase.thirdparty.com.google.common.collect.Sets;
final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private static final Logger LOG = LoggerFactory.getLogger(RSGroupInfoManagerImpl.class);
- private static final String REASSIGN_WAIT_INTERVAL_KEY = "hbase.rsgroup.reassign.wait";
- private static final long DEFAULT_REASSIGN_WAIT_INTERVAL = 30 * 1000L;
-
// Assigned before user tables
@VisibleForTesting
static final TableName RSGROUP_TABLE_NAME =
@@ -120,6 +120,9 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@VisibleForTesting
static final byte[] META_QUALIFIER_BYTES = Bytes.toBytes("i");
+ @VisibleForTesting
+ static final String MIGRATE_THREAD_NAME = "Migrate-RSGroup-Tables";
+
private static final byte[] ROW_KEY = { 0 };
/** Table descriptor for <code>hbase:rsgroup</code> catalog table */
@@ -140,7 +143,30 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// There two Maps are immutable and wholesale replaced on each modification
// so are safe to access concurrently. See class comment.
- private volatile Map<String, RSGroupInfo> rsGroupMap = Collections.emptyMap();
+ private static final class RSGroupInfoHolder {
+ final ImmutableMap<String, RSGroupInfo> groupName2Group;
+ final ImmutableMap<TableName, RSGroupInfo> tableName2Group;
+
+ RSGroupInfoHolder() {
+ this(Collections.emptyMap());
+ }
+
+ RSGroupInfoHolder(Map<String, RSGroupInfo> rsGroupMap) {
+ ImmutableMap.Builder<String, RSGroupInfo> group2Name2GroupBuilder = ImmutableMap.builder();
+ ImmutableMap.Builder<TableName, RSGroupInfo> tableName2GroupBuilder = ImmutableMap.builder();
+ rsGroupMap.forEach((groupName, rsGroupInfo) -> {
+ group2Name2GroupBuilder.put(groupName, rsGroupInfo);
+ if (!groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
+ rsGroupInfo.getTables()
+ .forEach(tableName -> tableName2GroupBuilder.put(tableName, rsGroupInfo));
+ }
+ });
+ this.groupName2Group = group2Name2GroupBuilder.build();
+ this.tableName2Group = tableName2GroupBuilder.build();
+ }
+ }
+
+ private volatile RSGroupInfoHolder holder = new RSGroupInfoHolder();
private final MasterServices masterServices;
private final AsyncClusterConnection conn;
@@ -160,9 +186,10 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
private synchronized void init() throws IOException {
- refresh();
+ refresh(false);
serverEventsListenerThread.start();
masterServices.getServerManager().registerListener(serverEventsListenerThread);
+ migrate();
}
static RSGroupInfoManager getInstance(MasterServices master) throws IOException {
@@ -179,6 +206,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public synchronized void addRSGroup(RSGroupInfo rsGroupInfo) throws IOException {
checkGroupName(rsGroupInfo.getName());
+ Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
if (rsGroupMap.get(rsGroupInfo.getName()) != null ||
rsGroupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
throw new DoNotRetryIOException("Group already exists: " + rsGroupInfo.getName());
@@ -235,7 +263,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
dst.addServer(el);
}
- Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
+ Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(holder.groupName2Group);
newGroupMap.put(src.getName(), src);
newGroupMap.put(dst.getName(), dst);
flushConfig(newGroupMap);
@@ -244,7 +272,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public RSGroupInfo getRSGroupOfServer(Address serverHostPort) throws IOException {
- for (RSGroupInfo info : rsGroupMap.values()) {
+ for (RSGroupInfo info : holder.groupName2Group.values()) {
if (info.containsServer(serverHostPort)) {
return info;
}
@@ -254,11 +282,12 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public RSGroupInfo getRSGroup(String groupName) {
- return rsGroupMap.get(groupName);
+ return holder.groupName2Group.get(groupName);
}
@Override
public synchronized void removeRSGroup(String groupName) throws IOException {
+ Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
if (!rsGroupMap.containsKey(groupName) || groupName.equals(RSGroupInfo.DEFAULT_GROUP)) {
throw new DoNotRetryIOException(
"Group " + groupName + " does not exist or is a reserved " + "group");
@@ -270,7 +299,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
@Override
public List<RSGroupInfo> listRSGroups() {
- return Lists.newArrayList(rsGroupMap.values());
+ return Lists.newArrayList(holder.groupName2Group.values());
}
@Override
@@ -298,7 +327,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
if (rsGroupInfos.size() > 0) {
- Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
+ Map<String, RSGroupInfo> newGroupMap = Maps.newHashMap(holder.groupName2Group);
newGroupMap.putAll(rsGroupInfos);
flushConfig(newGroupMap);
}
@@ -349,9 +378,90 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
return RSGroupInfoList;
}
- @Override
- public void refresh() throws IOException {
- refresh(false);
+ private void migrate(Collection<RSGroupInfo> groupList) {
+ TableDescriptors tds = masterServices.getTableDescriptors();
+ for (RSGroupInfo groupInfo : groupList) {
+ if (groupInfo.getName().equals(RSGroupInfo.DEFAULT_GROUP)) {
+ continue;
+ }
+ SortedSet<TableName> failedTables = new TreeSet<>();
+ for (TableName tableName : groupInfo.getTables()) {
+ LOG.debug("Migrating {} in group {}", tableName, groupInfo.getName());
+ TableDescriptor oldTd;
+ try {
+ oldTd = tds.get(tableName);
+ } catch (IOException e) {
+ LOG.warn("Failed to migrate {} in group {}", tableName, groupInfo.getName(), e);
+ failedTables.add(tableName);
+ continue;
+ }
+ if (oldTd == null) {
+ continue;
+ }
+ if (oldTd.getRegionServerGroup().isPresent()) {
+ // either we have already migrated it or that user has set the rs group using the new
+ // code which will set the group directly on table descriptor, skip.
+ LOG.debug("Skip migrating {} since it is already in group {}", tableName,
+ oldTd.getRegionServerGroup().get());
+ continue;
+ }
+ TableDescriptor newTd = TableDescriptorBuilder.newBuilder(oldTd)
+ .setRegionServerGroup(groupInfo.getName()).build();
+ // This is a bit tricky. Since we know that the region server group config in
+ // TableDescriptor will only be used at master side, it is fine to just update the table
+ // descriptor on file system and also the cache, without reopening all the regions. This
+ // will be much faster than the normal modifyTable. And when upgrading, we will update
+ // master first and then region server, so after all the region servers has been reopened,
+ // the new TableDescriptor will be loaded.
+ try {
+ tds.add(newTd);
+ } catch (IOException e) {
+ LOG.warn("Failed to migrate {} in group {}", tableName, groupInfo.getName(), e);
+ failedTables.add(tableName);
+ continue;
+ }
+ }
+ LOG.debug("Done migrating {}, failed tables {}", groupInfo.getName(), failedTables);
+ synchronized (RSGroupInfoManagerImpl.this) {
+ Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
+ RSGroupInfo currentInfo = rsGroupMap.get(groupInfo.getName());
+ if (currentInfo != null) {
+ RSGroupInfo newInfo =
+ new RSGroupInfo(currentInfo.getName(), currentInfo.getServers(), failedTables);
+ Map<String, RSGroupInfo> newGroupMap = new HashMap<>(rsGroupMap);
+ newGroupMap.put(groupInfo.getName(), newInfo);
+ try {
+ flushConfig(newGroupMap);
+ } catch (IOException e) {
+ LOG.warn("Failed to persist rs group {}", newInfo.getName(), e);
+ }
+ }
+ }
+ }
+ }
+
+ // Migrate the table rs group info from RSGroupInfo into the table descriptor
+ // Notice that we do not want to block the initialize so this will be done in background, and
+ // during the migrating, the rs group info maybe incomplete and cause region to be misplaced.
+ private void migrate() {
+ Thread migrateThread = new Thread(MIGRATE_THREAD_NAME) {
+
+ @Override
+ public void run() {
+ LOG.info("Start migrating table rs group config");
+ while (!masterServices.isStopped()) {
+ Collection<RSGroupInfo> groups = holder.groupName2Group.values();
+ boolean hasTables = groups.stream().anyMatch(r -> !r.getTables().isEmpty());
+ if (!hasTables) {
+ break;
+ }
+ migrate(groups);
+ }
+ LOG.info("Done migrating table rs group info");
+ }
+ };
+ migrateThread.setDaemon(true);
+ migrateThread.start();
}
/**
@@ -381,7 +491,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
newGroupMap.put(group.getName(), group);
}
resetRSGroupMap(newGroupMap);
- updateCacheOfRSGroups(rsGroupMap.keySet());
+ updateCacheOfRSGroups(newGroupMap.keySet());
}
private void flushConfigTable(Map<String, RSGroupInfo> groupMap) throws IOException {
@@ -411,20 +521,20 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
}
private synchronized void flushConfig() throws IOException {
- flushConfig(this.rsGroupMap);
+ flushConfig(holder.groupName2Group);
}
private synchronized void flushConfig(Map<String, RSGroupInfo> newGroupMap) throws IOException {
// For offline mode persistence is still unavailable
// We're refreshing in-memory state but only for servers in default group
if (!isOnline()) {
- if (newGroupMap == this.rsGroupMap) {
+ if (newGroupMap == holder.groupName2Group) {
// When newGroupMap is this.rsGroupMap itself,
// do not need to check default group and other groups as followed
return;
}
- Map<String, RSGroupInfo> oldGroupMap = Maps.newHashMap(rsGroupMap);
+ Map<String, RSGroupInfo> oldGroupMap = Maps.newHashMap(holder.groupName2Group);
RSGroupInfo oldDefaultGroup = oldGroupMap.remove(RSGroupInfo.DEFAULT_GROUP);
RSGroupInfo newDefaultGroup = newGroupMap.remove(RSGroupInfo.DEFAULT_GROUP);
if (!oldGroupMap.equals(newGroupMap) /* compare both tables and servers in other groups */ ||
@@ -438,7 +548,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Refresh rsGroupMap
// according to the inputted newGroupMap (an updated copy of rsGroupMap)
- rsGroupMap = newGroupMap;
+ this.holder = new RSGroupInfoHolder(newGroupMap);
// Do not need to update tableMap
// because only the update on servers in default group is allowed above,
@@ -495,8 +605,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
* Make changes visible. Caller must be synchronized on 'this'.
*/
private void resetRSGroupMap(Map<String, RSGroupInfo> newRSGroupMap) {
- // Make maps Immutable.
- this.rsGroupMap = Collections.unmodifiableMap(newRSGroupMap);
+ this.holder = new RSGroupInfoHolder(newRSGroupMap);
}
/**
@@ -549,6 +658,7 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
// Called by ServerEventsListenerThread. Synchronize on this because redoing
// the rsGroupMap then writing it out.
private synchronized void updateDefaultServers(SortedSet<Address> servers) {
+ Map<String, RSGroupInfo> rsGroupMap = holder.groupName2Group;
RSGroupInfo info = rsGroupMap.get(RSGroupInfo.DEFAULT_GROUP);
RSGroupInfo newInfo = new RSGroupInfo(info.getName(), servers);
HashMap<String, RSGroupInfo> newGroupMap = Maps.newHashMap(rsGroupMap);
@@ -647,6 +757,8 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
online = true;
// flush any inconsistencies between ZK and HTable
RSGroupInfoManagerImpl.this.flushConfig();
+ // migrate after we are online.
+ migrate();
return true;
} catch (Exception e) {
LOG.warn("Failed to perform check", e);
@@ -725,4 +837,10 @@ final class RSGroupInfoManagerImpl implements RSGroupInfoManager {
throw new ConstraintException("RSGroup name should only contain alphanumeric characters");
}
}
+
+
+ @Override
+ public RSGroupInfo getRSGroupForTable(TableName tableName) throws IOException {
+ return holder.tableName2Group.get(tableName);
+ }
}
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
index a08d236..af30049 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java
@@ -34,12 +34,11 @@ final class RSGroupUtil {
private RSGroupUtil() {
}
- @FunctionalInterface
- private interface GetRSGroup {
- RSGroupInfo get(String groupName) throws IOException;
- }
-
- private static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, GetRSGroup getter,
+ /**
+ * Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
+ * from the {@link NamespaceDescriptor}. If still not present, return empty.
+ */
+ static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupInfoManager manager,
TableName tableName) throws IOException {
TableDescriptor td = master.getTableDescriptors().get(tableName);
if (td == null) {
@@ -47,11 +46,17 @@ final class RSGroupUtil {
}
Optional<String> optGroupNameOfTable = td.getRegionServerGroup();
if (optGroupNameOfTable.isPresent()) {
- RSGroupInfo group = getter.get(optGroupNameOfTable.get());
+ RSGroupInfo group = manager.getRSGroup(optGroupNameOfTable.get());
if (group != null) {
return Optional.of(group);
}
}
+ // for backward compatible, where we may still have table configs in the RSGroupInfo after
+ // upgrading when migrating is still on-going.
+ RSGroupInfo groupFromOldRSGroupInfo = manager.getRSGroupForTable(tableName);
+ if (groupFromOldRSGroupInfo != null) {
+ return Optional.of(groupFromOldRSGroupInfo);
+ }
ClusterSchema clusterSchema = master.getClusterSchema();
if (clusterSchema == null) {
if (TableName.isMetaTableName(tableName)) {
@@ -67,25 +72,7 @@ final class RSGroupUtil {
if (groupNameOfNs == null) {
return Optional.empty();
}
- return Optional.ofNullable(getter.get(groupNameOfNs));
- }
-
- /**
- * Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
- * from the {@link NamespaceDescriptor}. If still not present, return empty.
- */
- static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupInfoManager manager,
- TableName tableName) throws IOException {
- return getRSGroupInfo(master, manager::getRSGroup, tableName);
- }
-
- /**
- * Will try to get the rsgroup from {@link TableDescriptor} first, and then try to get the rsgroup
- * from the {@link NamespaceDescriptor}. If still not present, return empty.
- */
- static Optional<RSGroupInfo> getRSGroupInfo(MasterServices master, RSGroupAdmin admin,
- TableName tableName) throws IOException {
- return getRSGroupInfo(master, admin::getRSGroupInfo, tableName);
+ return Optional.ofNullable(manager.getRSGroup(groupNameOfNs));
}
/**
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java
new file mode 100644
index 0000000..f61e18a
--- /dev/null
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestMigrateRSGroupInfo.java
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.rsgroup;
+
+import static org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.META_FAMILY_BYTES;
+import static org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.META_QUALIFIER_BYTES;
+import static org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl.RSGROUP_TABLE_NAME;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.util.concurrent.CountDownLatch;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.protobuf.generated.RSGroupProtos;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.zookeeper.KeeperException;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * Testcase for HBASE-22819
+ */
+@Category({ MediumTests.class })
+public class TestMigrateRSGroupInfo extends TestRSGroupsBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+ HBaseClassTestRule.forClass(TestMigrateRSGroupInfo.class);
+
+ private static String TABLE_NAME_PREFIX = "Table_";
+
+ private static int NUM_TABLES = 10;
+
+ private static byte[] FAMILY = Bytes.toBytes("family");
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ TEST_UTIL.getConfiguration().setClass(HConstants.MASTER_IMPL, HMasterForTest.class,
+ HMaster.class);
+ setUpTestBeforeClass();
+ for (int i = 0; i < NUM_TABLES; i++) {
+ TEST_UTIL.createTable(TableName.valueOf(TABLE_NAME_PREFIX + i), FAMILY);
+ }
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ tearDownAfterClass();
+ }
+
+ private static CountDownLatch RESUME = new CountDownLatch(1);
+
+ public static final class HMasterForTest extends HMaster {
+
+ public HMasterForTest(Configuration conf) throws IOException, KeeperException {
+ super(conf);
+ }
+
+ @Override
+ public TableDescriptors getTableDescriptors() {
+ if (RESUME != null) {
+ for (StackTraceElement element : Thread.currentThread().getStackTrace()) {
+ if (element.getClassName().contains("RSGroupInfoManagerImpl")) {
+ try {
+ RESUME.await();
+ } catch (InterruptedException e) {
+ }
+ RESUME = null;
+ break;
+ }
+ }
+ }
+ return super.getTableDescriptors();
+ }
+ }
+
+ @Test
+ public void testMigrate() throws IOException, InterruptedException {
+ String groupName = name.getMethodName();
+ addGroup(groupName, TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().size() - 1);
+ RSGroupInfo rsGroupInfo = rsGroupAdmin.getRSGroupInfo(groupName);
+ assertTrue(rsGroupInfo.getTables().isEmpty());
+ for (int i = 0; i < NUM_TABLES; i++) {
+ rsGroupInfo.addTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ }
+ try (Table table = TEST_UTIL.getConnection().getTable(RSGROUP_TABLE_NAME)) {
+ RSGroupProtos.RSGroupInfo proto = ProtobufUtil.toProtoGroupInfo(rsGroupInfo);
+ Put p = new Put(Bytes.toBytes(rsGroupInfo.getName()));
+ p.addColumn(META_FAMILY_BYTES, META_QUALIFIER_BYTES, proto.toByteArray());
+ table.put(p);
+ }
+ TEST_UTIL.getMiniHBaseCluster().stopMaster(0).join();
+ RESUME = new CountDownLatch(1);
+ TEST_UTIL.getMiniHBaseCluster().startMaster();
+
+ // wait until we can get the rs group info for a table
+ TEST_UTIL.waitFor(30000, () -> {
+ try {
+ rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + 0));
+ return true;
+ } catch (IOException e) {
+ return false;
+ }
+ });
+ // confirm that before migrating, we could still get the correct rs group for a table.
+ for (int i = 0; i < NUM_TABLES; i++) {
+ RSGroupInfo info =
+ rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ assertEquals(rsGroupInfo.getName(), info.getName());
+ assertEquals(NUM_TABLES, info.getTables().size());
+ }
+ RESUME.countDown();
+ TEST_UTIL.waitFor(60000, () -> {
+ for (int i = 0; i < NUM_TABLES; i++) {
+ TableDescriptor td;
+ try {
+ td = TEST_UTIL.getAdmin().getDescriptor(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ } catch (IOException e) {
+ return false;
+ }
+ if (!rsGroupInfo.getName().equals(td.getRegionServerGroup().orElse(null))) {
+ return false;
+ }
+ }
+ return true;
+ });
+ // make sure that we persist the result to hbase, where we delete all the tables in the rs
+ // group.
+ TEST_UTIL.waitFor(30000, () -> {
+ try (Table table = TEST_UTIL.getConnection().getTable(RSGROUP_TABLE_NAME)) {
+ Result result = table.get(new Get(Bytes.toBytes(rsGroupInfo.getName())));
+ RSGroupProtos.RSGroupInfo proto = RSGroupProtos.RSGroupInfo
+ .parseFrom(result.getValue(META_FAMILY_BYTES, META_QUALIFIER_BYTES));
+ RSGroupInfo gi = ProtobufUtil.toGroupInfo(proto);
+ return gi.getTables().isEmpty();
+ }
+ });
+ // make sure that the migrate thread has quit.
+ TEST_UTIL.waitFor(30000, () -> Thread.getAllStackTraces().keySet().stream()
+ .noneMatch(t -> t.getName().equals(RSGroupInfoManagerImpl.MIGRATE_THREAD_NAME)));
+ // make sure we could still get the correct rs group info after migration
+ for (int i = 0; i < NUM_TABLES; i++) {
+ RSGroupInfo info =
+ rsGroupAdmin.getRSGroupInfoOfTable(TableName.valueOf(TABLE_NAME_PREFIX + i));
+ assertEquals(rsGroupInfo.getName(), info.getName());
+ assertEquals(NUM_TABLES, info.getTables().size());
+ }
+ }
+}
diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
index b91cd5e..b9885be 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java
@@ -71,7 +71,7 @@ public abstract class TestRSGroupsBase {
protected final static Random rand = new Random();
//shared, cluster type specific
- protected static HBaseTestingUtility TEST_UTIL;
+ protected static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
protected static Admin admin;
protected static HBaseCluster cluster;
protected static RSGroupAdminClient rsGroupAdmin;
@@ -90,7 +90,6 @@ public abstract class TestRSGroupsBase {
protected TableName tableName;
public static void setUpTestBeforeClass() throws Exception {
- TEST_UTIL = new HBaseTestingUtility();
TEST_UTIL.getConfiguration().setFloat(
"hbase.master.balancer.stochastic.tableSkewCost", 6000);
TEST_UTIL.getConfiguration().set(
[hbase] 12/14: HBASE-23232 Remove rsgroup profile from pom.xml of
hbase-assembly (#779)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch HBASE-22514
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 0201d4f2b16f43fb4ee06d3bfe7dcc08439c97d1
Author: Guangxu Cheng <gu...@gmail.com>
AuthorDate: Thu Oct 31 03:01:09 2019 +0800
HBASE-23232 Remove rsgroup profile from pom.xml of hbase-assembly (#779)
---
hbase-assembly/pom.xml | 17 -----------------
1 file changed, 17 deletions(-)
diff --git a/hbase-assembly/pom.xml b/hbase-assembly/pom.xml
index 9096e09..f9e4431 100644
--- a/hbase-assembly/pom.xml
+++ b/hbase-assembly/pom.xml
@@ -323,21 +323,4 @@
<type>pom</type>
</dependency>
</dependencies>
- <profiles>
- <profile>
- <id>rsgroup</id>
- <activation>
- <property>
- <name>!skip-rsgroup</name>
- </property>
- </activation>
- <dependencies>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-rsgroup</artifactId>
- <version>${project.version}</version>
- </dependency>
- </dependencies>
- </profile>
- </profiles>
</project>