You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/03/31 15:18:55 UTC
[GitHub] [hbase-operator-tools] BukrosSzabolcs opened a new pull request
#56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
BukrosSzabolcs opened a new pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56
…merging multiple adjacent regions until a desired number of regions is reached.
This is a continuation of #46 and was opened because @wchevreuil is temporarily unavailable and I could not add my changes to the original PR.
I made a few changes based on @joshelser 's feedback and a small fix in the merging for cycle's break condition.
@busbey on the open questions, would you think the CLI option a must for this first version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser commented on a change in pull
request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that
allows for...
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56#discussion_r404329830
##########
File path: hbase-tools/src/main/java/org/apache/hbase/RegionsMerger.java
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import static org.apache.hadoop.hbase.HConstants.CATALOG_FAMILY;
+import static org.apache.hadoop.hbase.HConstants.REGIONINFO_QUALIFIER;
+import static org.apache.hadoop.hbase.HConstants.STATE_QUALIFIER;
+import static org.apache.hadoop.hbase.TableName.META_TABLE_NAME;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.LongAdder;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * HBase maintenance tool for merging regions of a specific table, until a target number of regions
+ * for the table is reached, or no more merges can complete due to limit in resulting merged
+ * region size.
+ */
+public class RegionsMerger extends Configured implements org.apache.hadoop.util.Tool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(RegionsMerger.class.getName());
+ public static final String RESULTING_REGION_UPPER_MARK = "hbase.tools.merge.upper.mark";
+ public static final String SLEEP = "hbase.tools.merge.sleep";
+ public static final String MAX_ROUNDS_IDDLE = "hbase.tools.max.iterations.blocked";
Review comment:
nit: IDDLE
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser closed pull request #56:
HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
Posted by GitBox <gi...@apache.org>.
joshelser closed pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser commented on a change in pull
request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that
allows for...
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56#discussion_r404329262
##########
File path: hbase-tools/pom.xml
##########
@@ -0,0 +1,202 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ /**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ -->
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <artifactId>hbase-operator-tools</artifactId>
+ <groupId>org.apache.hbase.operator.tools</groupId>
+ <version>1.1.0-SNAPSHOT</version>
+ <relativePath>..</relativePath>
+ </parent>
+
+
+ <artifactId>hbase-tools</artifactId>
+ <name>Apache HBase - HBase Tools</name>
+ <description>Utility Maintenance tools for HBase 2+</description>
+ <properties>
+ <hbase-thirdparty.version>2.2.1</hbase-thirdparty.version>
+ <log4j2.version>2.11.1</log4j2.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
+
+ <!--We want to use the shaded client but for testing, we need to rely on hbase-server.
+ HBASE-15666 is about how shaded-client and hbase-server won't work together.
+ TODO: Fix.-->
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-server</artifactId>
+ <version>${hbase.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-shaded-testing-util</artifactId>
Review comment:
Should this be a test scope dependency?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser commented on a change in pull
request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that
allows for...
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56#discussion_r404330577
##########
File path: hbase-tools/src/main/java/org/apache/hbase/RegionsMerger.java
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import static org.apache.hadoop.hbase.HConstants.CATALOG_FAMILY;
+import static org.apache.hadoop.hbase.HConstants.REGIONINFO_QUALIFIER;
+import static org.apache.hadoop.hbase.HConstants.STATE_QUALIFIER;
+import static org.apache.hadoop.hbase.TableName.META_TABLE_NAME;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.LongAdder;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * HBase maintenance tool for merging regions of a specific table, until a target number of regions
+ * for the table is reached, or no more merges can complete due to limit in resulting merged
+ * region size.
+ */
+public class RegionsMerger extends Configured implements org.apache.hadoop.util.Tool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(RegionsMerger.class.getName());
+ public static final String RESULTING_REGION_UPPER_MARK = "hbase.tools.merge.upper.mark";
+ public static final String SLEEP = "hbase.tools.merge.sleep";
+ public static final String MAX_ROUNDS_IDDLE = "hbase.tools.max.iterations.blocked";
+
+ private final Configuration conf;
+ private final FileSystem fs;
+ private final double resultSizeThreshold;
+ private final int sleepBetweenCycles;
+ private final long maxRoundsStuck;
+
+ public RegionsMerger(Configuration conf) throws IOException {
+ this.conf = conf;
+ Path basePath = new Path(conf.get(HConstants.HBASE_DIR));
+ fs = basePath.getFileSystem(conf);
+ resultSizeThreshold = this.conf.getDouble(RESULTING_REGION_UPPER_MARK, 0.9) *
+ this.conf.getLong(HConstants.HREGION_MAX_FILESIZE, HConstants.DEFAULT_MAX_FILE_SIZE);
+ sleepBetweenCycles = this.conf.getInt(SLEEP, 2000);
+ this.maxRoundsStuck = this.conf.getInt(MAX_ROUNDS_IDDLE, 10);
+ }
+
+ private Path getTablePath(TableName table){
+ Path basePath = new Path(conf.get(HConstants.HBASE_DIR));
+ basePath = new Path(basePath, "data");
+ Path tablePath = new Path(basePath, table.getNamespaceAsString());
+ return new Path(tablePath, table.getNameAsString());
+ }
+
+ private long sumSizeInFS(Path parentPath) throws IOException {
+ long size = 0;
+ FileStatus[] files = this.fs.listStatus(parentPath);
+ for(FileStatus f : files) {
+ if(f.isFile()) {
+ size += f.getLen();
+ } else if(f.isDirectory()) {
+ size += sumSizeInFS(f.getPath());
+ }
+ }
+ return size;
+ }
+
+ private List<RegionInfo> getOpenRegions(Connection connection, TableName table) throws Exception {
+ List<RegionInfo> regions = new ArrayList<>();
+ Table metaTbl = connection.getTable(META_TABLE_NAME);
+ String tblName = table.getNameAsString();
+ RowFilter rowFilter = new RowFilter(CompareOperator.EQUAL,
+ new SubstringComparator(tblName+","));
+ SingleColumnValueFilter colFilter = new SingleColumnValueFilter(CATALOG_FAMILY,
+ STATE_QUALIFIER, CompareOperator.EQUAL, Bytes.toBytes("OPEN"));
+ Scan scan = new Scan();
+ FilterList filter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ filter.addFilter(rowFilter);
+ filter.addFilter(colFilter);
+ scan.setFilter(filter);
+ ResultScanner rs = metaTbl.getScanner(scan);
Review comment:
try/finally to close the `ResultScanner` please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser commented on a change in pull
request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that
allows for...
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56#discussion_r404333327
##########
File path: hbase-tools/src/test/java/org/apache/hbase/TestRegionsMerger.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+
+public class TestRegionsMerger {
+ private final static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
+ private static final TableName TABLE_NAME =
+ TableName.valueOf(TestRegionsMerger.class.getSimpleName());
+ private static final byte[] family = Bytes.toBytes("f");
+ private Table table;
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+ TEST_UTIL.getConfiguration().set(HConstants.HREGION_MAX_FILESIZE,
+ Long.toString(1024*1024*3));
+ TEST_UTIL.startMiniCluster(3);
+ }
+
+ @AfterClass
+ public static void afterClass() throws Exception {
+ TEST_UTIL.shutdownMiniCluster();
+ }
+
+ @Before
+ public void setup() throws Exception {
+ table = TEST_UTIL.createMultiRegionTable(TABLE_NAME, family, 15);
+ TEST_UTIL.waitUntilAllRegionsAssigned(TABLE_NAME);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ TEST_UTIL.deleteTable(TABLE_NAME);
+ }
+
+ @Test
+ public void testMergeRegionsCanMergeToTarget() throws Exception {
+ TEST_UTIL.getConfiguration().setInt(RegionsMerger.MAX_ROUNDS_IDDLE, 10);
+ RegionsMerger merger = new RegionsMerger(TEST_UTIL.getConfiguration());
+ merger.mergeRegions(TABLE_NAME.getNameWithNamespaceInclAsString(), 3);
+ List<RegionInfo> result = TEST_UTIL.getAdmin().getRegions(TABLE_NAME);
+ assertEquals(3, result.size());
Review comment:
For all of the tests, can you please add a verification step to make sure that we can still read all records in the table?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] joshelser commented on a change in pull
request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that
allows for...
Posted by GitBox <gi...@apache.org>.
joshelser commented on a change in pull request #56: HBASE-23562 [operator tools] Add a RegionsMerge tool that allows for...
URL: https://github.com/apache/hbase-operator-tools/pull/56#discussion_r404332777
##########
File path: hbase-tools/src/main/java/org/apache/hbase/RegionsMerger.java
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import static org.apache.hadoop.hbase.HConstants.CATALOG_FAMILY;
+import static org.apache.hadoop.hbase.HConstants.REGIONINFO_QUALIFIER;
+import static org.apache.hadoop.hbase.HConstants.STATE_QUALIFIER;
+import static org.apache.hadoop.hbase.TableName.META_TABLE_NAME;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.LongAdder;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.filter.FilterList;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * HBase maintenance tool for merging regions of a specific table, until a target number of regions
+ * for the table is reached, or no more merges can complete due to limit in resulting merged
+ * region size.
+ */
+public class RegionsMerger extends Configured implements org.apache.hadoop.util.Tool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(RegionsMerger.class.getName());
+ public static final String RESULTING_REGION_UPPER_MARK = "hbase.tools.merge.upper.mark";
+ public static final String SLEEP = "hbase.tools.merge.sleep";
+ public static final String MAX_ROUNDS_IDDLE = "hbase.tools.max.iterations.blocked";
+
+ private final Configuration conf;
+ private final FileSystem fs;
+ private final double resultSizeThreshold;
+ private final int sleepBetweenCycles;
+ private final long maxRoundsStuck;
+
+ public RegionsMerger(Configuration conf) throws IOException {
+ this.conf = conf;
+ Path basePath = new Path(conf.get(HConstants.HBASE_DIR));
+ fs = basePath.getFileSystem(conf);
+ resultSizeThreshold = this.conf.getDouble(RESULTING_REGION_UPPER_MARK, 0.9) *
+ this.conf.getLong(HConstants.HREGION_MAX_FILESIZE, HConstants.DEFAULT_MAX_FILE_SIZE);
+ sleepBetweenCycles = this.conf.getInt(SLEEP, 2000);
+ this.maxRoundsStuck = this.conf.getInt(MAX_ROUNDS_IDDLE, 10);
+ }
+
+ private Path getTablePath(TableName table){
+ Path basePath = new Path(conf.get(HConstants.HBASE_DIR));
+ basePath = new Path(basePath, "data");
+ Path tablePath = new Path(basePath, table.getNamespaceAsString());
+ return new Path(tablePath, table.getNameAsString());
+ }
+
+ private long sumSizeInFS(Path parentPath) throws IOException {
+ long size = 0;
+ FileStatus[] files = this.fs.listStatus(parentPath);
+ for(FileStatus f : files) {
+ if(f.isFile()) {
+ size += f.getLen();
+ } else if(f.isDirectory()) {
+ size += sumSizeInFS(f.getPath());
+ }
+ }
+ return size;
+ }
+
+ private List<RegionInfo> getOpenRegions(Connection connection, TableName table) throws Exception {
+ List<RegionInfo> regions = new ArrayList<>();
+ Table metaTbl = connection.getTable(META_TABLE_NAME);
+ String tblName = table.getNameAsString();
+ RowFilter rowFilter = new RowFilter(CompareOperator.EQUAL,
+ new SubstringComparator(tblName+","));
+ SingleColumnValueFilter colFilter = new SingleColumnValueFilter(CATALOG_FAMILY,
+ STATE_QUALIFIER, CompareOperator.EQUAL, Bytes.toBytes("OPEN"));
+ Scan scan = new Scan();
+ FilterList filter = new FilterList(FilterList.Operator.MUST_PASS_ALL);
+ filter.addFilter(rowFilter);
+ filter.addFilter(colFilter);
+ scan.setFilter(filter);
+ ResultScanner rs = metaTbl.getScanner(scan);
+ Result r;
+ while ((r = rs.next()) != null) {
+ RegionInfo region = RegionInfo.parseFrom(r.getValue(CATALOG_FAMILY, REGIONINFO_QUALIFIER));
+ regions.add(region);
+ }
+ rs.close();
+ return regions;
+ }
+
+ private boolean canMerge(Path path, RegionInfo region1, RegionInfo region2,
+ Collection<Pair<RegionInfo, RegionInfo>> alreadyMerging) throws IOException {
+ if(alreadyMerging.stream().anyMatch(regionPair ->
+ region1.equals(regionPair.getFirst()) ||
+ region2.equals(regionPair.getFirst()) ||
+ region1.equals(regionPair.getSecond()) ||
+ region2.equals(regionPair.getSecond()))){
+ return false;
+ }
+ if (RegionInfo.areAdjacent(region1, region2)) {
+ long size1 = sumSizeInFS(new Path(path, region1.getEncodedName()));
+ long size2 = sumSizeInFS(new Path(path, region2.getEncodedName()));
+ boolean mergeable = (resultSizeThreshold > (size1 + size2));
+ if (!mergeable) {
+ LOG.warn("Not merging regions {} and {} because resulting region size would get close to " +
+ "the {} limit. {} total size: {}; {} total size:{}", region1.getEncodedName(),
+ region2.getEncodedName(), resultSizeThreshold, region1.getEncodedName(), size1,
+ region2.getEncodedName(), size2);
+ }
+ return mergeable;
+ } else {
+ LOG.warn(
+ "WARNING: Can't merge regions {} and {} because those are not adjacent.",
+ region1.getEncodedName(),
+ region2.getEncodedName());
+ return false;
+ }
+ }
+
+ public void mergeRegions(String tblName, int targetRegions) throws Exception {
+ TableName table = TableName.valueOf(tblName);
+ Path tableDir = getTablePath(table);
+ try(Connection conn = ConnectionFactory.createConnection(conf)) {
+ Admin admin = conn.getAdmin();
+ LongAdder counter = new LongAdder();
+ LongAdder lastTimeProgessed = new LongAdder();
+ //need to get all regions for the table, regardless of region state
+ List<RegionInfo> regions = admin.getRegions(table);
+ Map<Future, Pair<RegionInfo, RegionInfo>> regionsMerging = new HashMap<>();
+ long roundsNoProgress = 0;
+ while (regions.size() > targetRegions) {
+ LOG.info("Iteration: {}", counter);
+ RegionInfo previous = null;
+ int regionSize = regions.size();
+ LOG.info("Attempting to merge {} regions to reach the target {} ...", regionSize, targetRegions);
+ //to request merge, regions must be OPEN, though
+ regions = getOpenRegions(conn, table);
+ for (RegionInfo current : regions) {
+ if (!current.isSplit()) {
+ if (previous != null && canMerge(tableDir, previous, current, regionsMerging.values())) {
+ Future f = admin.mergeRegionsAsync(current.getEncodedNameAsBytes(),
+ previous.getEncodedNameAsBytes(), true);
+ Pair<RegionInfo, RegionInfo> regionPair = new Pair<>(previous, current);
+ regionsMerging.put(f,regionPair);
+ previous = null;
+ if ((regionSize - regionsMerging.size()) <= targetRegions) {
+ break;
+ }
+ } else {
+ previous = current;
+ }
+ }
+ else{
+ LOG.debug("Skipping split region: {}", current.getEncodedName());
+ }
+ }
+ counter.increment();
+ LOG.info("Sleeping for {} seconds before next iteration...", (sleepBetweenCycles/1000));
+ Thread.sleep(sleepBetweenCycles);
+ regionsMerging.forEach((f, currentPair)-> {
+ if (f.isDone()) {
+ LOG.info("Merged regions {} and {} together.",
+ currentPair.getFirst().getEncodedName(),
+ currentPair.getSecond().getEncodedName());
+ regionsMerging.remove(f);
+ lastTimeProgessed.reset();
+ lastTimeProgessed.add(counter.longValue());
+ } else {
+ LOG.warn("Merge of regions {} and {} isn't completed yet.",
+ currentPair.getFirst(),
+ currentPair.getSecond());
+ }
+ });
+ roundsNoProgress = counter.longValue() - lastTimeProgessed.longValue();
+ if(roundsNoProgress == this.maxRoundsStuck){
+ LOG.warn("Reached {} iterations without progressing with new merges. Aborting...",
+ roundsNoProgress);
+ break;
+ }
+
+ //again, get all regions, regardless of the state,
+ // in order to avoid breaking the loop prematurely
+ regions = admin.getRegions(table);
+ }
+ }
+ }
+
+ @Override
+ public int run(String[] args) {
+ if(args.length!=2){
+ LOG.error("Wrong number of arguments. "
+ + "Arguments are: <TABLE_NAME> <TARGET_NUMBER_OF_REGIONS>");
+ return 1;
+ }
+ try {
+ this.mergeRegions(args[0], Integer.parseInt(args[1]));
+ } catch(Exception e){
+ LOG.error("Merging regions failed:", e);
+ return 2;
+ }
+ return 0;
+ }
+
+ public static void main(String [] args) throws Exception {
+ Configuration conf = HBaseConfiguration.create();
+ int errCode = org.apache.hadoop.util.ToolRunner.run(new RegionsMerger(conf), args);
Review comment:
nit: is there a reason we're using the fully qualified class name for ToolRunner here? (vs. using an import)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services