You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tephra.apache.org by gokulavasan <gi...@git.apache.org> on 2017/02/07 04:08:52 UTC
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
GitHub user gokulavasan opened a pull request:
https://github.com/apache/incubator-tephra/pull/31
(TEPHRA-214) Tool to debug the state and progress of Invalid List Pruning
JIRA : https://issues.apache.org/jira/browse/TEPHRA-214
Summary: Introduce a debug tool to fetch data that might help with debugging the status and progress of invalid list transaction list pruning.
Three commands are supported:
timeregion <timestamp> ==> Will return TimeRegions at or before timestamp in millis
idleregions <numOfRegions> ==> Will return numOfRegions regions with lowest pruneUpperBounds. If -1 is provided, all pruneUpperBounds are returned.
pruneinfo <regionNameAsString> => Will return pruneUpperBound, compactionTimeStamp of the 'regionNameAsString' region.
This tool can be invoked with hbase classpath and tephra jars in the classpath.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gokulavasan/incubator-tephra feature/tephra-pruning-debug
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tephra/pull/31.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #31
----
commit dc4357824e8e48973cc5109c649aa8303f077233
Author: Gokul Gunasekaran <go...@cask.co>
Date: 2017-02-03T20:10:40Z
(TEPHRA-214) Tool to debug the state and progress of Invalid List Pruning
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #31: (TEPHRA-214) Tool to debug the state and progres...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on the issue:
https://github.com/apache/incubator-tephra/pull/31
@gokulavasan I took a pass over this, I have a few comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750194
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java ---
@@ -147,6 +147,7 @@ public long fetchPruneUpperBound(long time, long inactiveTransactionBound) throw
// Get all the current transactional regions
SortedSet<byte[]> transactionalRegions = getTransactionalRegions();
+ // TODO: Ignore empty
--- End diff --
This TODO needs to be handled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #31: (TEPHRA-214) Tool to debug the state and progres...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on the issue:
https://github.com/apache/incubator-tephra/pull/31
@poornachandra Ported changes to other modules. Please take a look. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750392
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
--- End diff --
Same here `idle-regions`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750355
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
--- End diff --
Can we make it `time-region`? It is more readable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99951182
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/HBaseTransactionPruningPlugin.java ---
@@ -147,6 +147,7 @@ public long fetchPruneUpperBound(long time, long inactiveTransactionBound) throw
// Get all the current transactional regions
SortedSet<byte[]> transactionalRegions = getTransactionalRegions();
+ // TODO: Ignore empty
--- End diff --
That TODO was meant for ignoring empty regions from the current transactional regions. I have removed that misleading TODO as part of the last commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99962992
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -129,17 +168,19 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
Result next;
while ((next = scanner.next()) != null) {
byte[] region = getRegionFromKey(next.getRow());
- if (regions.contains(region)) {
- byte[] timeBytes = next.getValue(FAMILY, PRUNE_UPPER_BOUND_COL);
- if (timeBytes != null) {
- long pruneUpperBoundRegion = Bytes.toLong(timeBytes);
- resultMap.put(region, pruneUpperBoundRegion);
+ if (regions == null || regions.contains(region)) {
+ Cell cell = next.getColumnLatestCell(FAMILY, PRUNE_UPPER_BOUND_COL);
+ if (cell != null) {
+ byte[] pruneUpperBoundBytes = CellUtil.cloneValue(cell);
+ long timestamp = cell.getTimestamp();
--- End diff --
What would be a good name for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99759929
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -102,11 +107,29 @@ public void savePruneUpperBoundForRegion(byte[] regionId, long pruneUpperBound)
* @throws IOException when not able to read the data from HBase
*/
public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
+ RegionPruneInfo regionPruneInfo = getPruneInfoForRegion(regionId);
+ return (regionPruneInfo == null) ? -1 : regionPruneInfo.getCompactionTimestamp();
--- End diff --
This has to return the `regionPruneInfo.getPruneUpperBound()`, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99760732
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -129,17 +168,19 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
Result next;
while ((next = scanner.next()) != null) {
byte[] region = getRegionFromKey(next.getRow());
- if (regions.contains(region)) {
- byte[] timeBytes = next.getValue(FAMILY, PRUNE_UPPER_BOUND_COL);
- if (timeBytes != null) {
- long pruneUpperBoundRegion = Bytes.toLong(timeBytes);
- resultMap.put(region, pruneUpperBoundRegion);
+ if (regions == null || regions.contains(region)) {
+ Cell cell = next.getColumnLatestCell(FAMILY, PRUNE_UPPER_BOUND_COL);
+ if (cell != null) {
+ byte[] pruneUpperBoundBytes = CellUtil.cloneValue(cell);
+ long timestamp = cell.getTimestamp();
--- End diff --
This is not compaction timestamp, this is when the prune upper bound was written to HBase. Although this can be an approximate value when the state table is available. Is the compaction timestamp really necessary for debugging?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99760211
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -120,6 +143,22 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
*/
public Map<byte[], Long> getPruneUpperBoundForRegions(SortedSet<byte[]> regions) throws IOException {
Map<byte[], Long> resultMap = new TreeMap<>(Bytes.BYTES_COMPARATOR);
+ List<RegionPruneInfo> regionPruneInfos = getPruneInfoForRegions(regions);
+ for (RegionPruneInfo regionPruneInfo : regionPruneInfos) {
+ resultMap.put(regionPruneInfo.getRegionName(), regionPruneInfo.getCompactionTimestamp());
--- End diff --
Same here, should be `regionPruneInfo.getPruneUpperBound()` instead of `regionPruneInfo.getCompactionTimestamp()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99949496
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -129,17 +168,19 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
Result next;
while ((next = scanner.next()) != null) {
byte[] region = getRegionFromKey(next.getRow());
- if (regions.contains(region)) {
- byte[] timeBytes = next.getValue(FAMILY, PRUNE_UPPER_BOUND_COL);
- if (timeBytes != null) {
- long pruneUpperBoundRegion = Bytes.toLong(timeBytes);
- resultMap.put(region, pruneUpperBoundRegion);
+ if (regions == null || regions.contains(region)) {
+ Cell cell = next.getColumnLatestCell(FAMILY, PRUNE_UPPER_BOUND_COL);
+ if (cell != null) {
+ byte[] pruneUpperBoundBytes = CellUtil.cloneValue(cell);
+ long timestamp = cell.getTimestamp();
--- End diff --
@poornachandra It might be useful information to have right? I can rename it to something else if you think compaction timestamp is misleading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99751362
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
--- End diff --
If `timeRegions` is returned as object from here, then it can be printed as JSON later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99761297
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
--- End diff --
`Prints out the transactional regions present in HBase at time...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750875
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
+ pw.println("pruneinfo regionNameAsString");
--- End diff --
Same here - `prune-info` and `region-name-as-string`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99751539
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
+ pw.println("pruneinfo regionNameAsString");
+ pw.println("Desc: Prints out the Pruning information for the region 'regionNameAsString'");
+ }
+
+ private boolean execute(String[] args) throws IOException {
+ try (PrintWriter pw = new PrintWriter(System.out)) {
+ if (args.length != 2) {
+ printUsage(pw);
+ return false;
+ }
+
+ String command = args[0];
+ String parameter = args[1];
+ if ("timeregion".equals(command)) {
+ Long time = Long.parseLong(parameter);
+ String timeRegion = getRegionsOnOrBeforeTime(time);
+ if (timeRegion != null) {
+ pw.println(timeRegion);
+ } else {
+ pw.println(String.format("No TimeRegion found for time %d or before that.", time));
+ }
+ return true;
+ } else if ("idleregions".equals(command)) {
+ Integer numRegions = Integer.parseInt(parameter);
+ Queue<RegionPruneInfo> regionPruneInfos = getIdleRegions(numRegions);
+ pw.println(GSON.toJson(regionPruneInfos));
+ return true;
+ } else if ("pruneinfo".endsWith(command)) {
+ RegionPruneInfo regionPruneInfo = getRegionPruneInfo(parameter);
+ if (regionPruneInfo != null) {
+ pw.println(regionPruneInfo.toString());
--- End diff --
This too can be printed as JSON
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99751506
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
+ pw.println("pruneinfo regionNameAsString");
+ pw.println("Desc: Prints out the Pruning information for the region 'regionNameAsString'");
+ }
+
+ private boolean execute(String[] args) throws IOException {
+ try (PrintWriter pw = new PrintWriter(System.out)) {
+ if (args.length != 2) {
+ printUsage(pw);
+ return false;
+ }
+
+ String command = args[0];
+ String parameter = args[1];
+ if ("timeregion".equals(command)) {
+ Long time = Long.parseLong(parameter);
+ String timeRegion = getRegionsOnOrBeforeTime(time);
+ if (timeRegion != null) {
+ pw.println(timeRegion);
+ } else {
+ pw.println(String.format("No TimeRegion found for time %d or before that.", time));
+ }
+ return true;
+ } else if ("idleregions".equals(command)) {
+ Integer numRegions = Integer.parseInt(parameter);
+ Queue<RegionPruneInfo> regionPruneInfos = getIdleRegions(numRegions);
+ pw.println(GSON.toJson(regionPruneInfos));
+ return true;
+ } else if ("pruneinfo".endsWith(command)) {
--- End diff --
This too can be `equals` instead of `endsWith`, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750124
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
+ pw.println("pruneinfo regionNameAsString");
+ pw.println("Desc: Prints out the Pruning information for the region 'regionNameAsString'");
+ }
+
+ private boolean execute(String[] args) throws IOException {
+ try (PrintWriter pw = new PrintWriter(System.out)) {
+ if (args.length != 2) {
+ printUsage(pw);
+ return false;
+ }
+
+ String command = args[0];
+ String parameter = args[1];
+ if ("timeregion".equals(command)) {
+ Long time = Long.parseLong(parameter);
+ String timeRegion = getRegionsOnOrBeforeTime(time);
+ if (timeRegion != null) {
+ pw.println(timeRegion);
+ } else {
+ pw.println(String.format("No TimeRegion found for time %d or before that.", time));
+ }
+ return true;
+ } else if ("idleregions".equals(command)) {
+ Integer numRegions = Integer.parseInt(parameter);
+ Queue<RegionPruneInfo> regionPruneInfos = getIdleRegions(numRegions);
+ pw.println(GSON.toJson(regionPruneInfos));
+ return true;
+ } else if ("pruneinfo".endsWith(command)) {
+ RegionPruneInfo regionPruneInfo = getRegionPruneInfo(parameter);
+ if (regionPruneInfo != null) {
+ pw.println(regionPruneInfo.toString());
+ } else {
+ pw.println(String.format("No prune region found for the region %s.", parameter));
+ }
+ return true;
+ } else {
+ pw.println(String.format("%s is not a valid command.", command));
+ printUsage(pw);
+ return false;
+ }
+ }
+ }
+
+ public static void main(String[] args) {
+ Configuration hConf = new Configuration();
--- End diff --
This should be `HBaseConfiguration.create()`, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #31: (TEPHRA-214) Tool to debug the state and progres...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on the issue:
https://github.com/apache/incubator-tephra/pull/31
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99751633
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
+ pw.println("pruneinfo regionNameAsString");
+ pw.println("Desc: Prints out the Pruning information for the region 'regionNameAsString'");
+ }
+
+ private boolean execute(String[] args) throws IOException {
+ try (PrintWriter pw = new PrintWriter(System.out)) {
+ if (args.length != 2) {
+ printUsage(pw);
+ return false;
+ }
+
+ String command = args[0];
+ String parameter = args[1];
+ if ("timeregion".equals(command)) {
+ Long time = Long.parseLong(parameter);
+ String timeRegion = getRegionsOnOrBeforeTime(time);
+ if (timeRegion != null) {
+ pw.println(timeRegion);
+ } else {
+ pw.println(String.format("No TimeRegion found for time %d or before that.", time));
+ }
+ return true;
+ } else if ("idleregions".equals(command)) {
+ Integer numRegions = Integer.parseInt(parameter);
+ Queue<RegionPruneInfo> regionPruneInfos = getIdleRegions(numRegions);
+ pw.println(GSON.toJson(regionPruneInfos));
+ return true;
+ } else if ("pruneinfo".endsWith(command)) {
+ RegionPruneInfo regionPruneInfo = getRegionPruneInfo(parameter);
+ if (regionPruneInfo != null) {
+ pw.println(regionPruneInfo.toString());
+ } else {
+ pw.println(String.format("No prune region found for the region %s.", parameter));
--- End diff --
`No prune region found` => `No prune info found`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra issue #31: (TEPHRA-214) Tool to debug the state and progres...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on the issue:
https://github.com/apache/incubator-tephra/pull/31
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750467
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
--- End diff --
Also `numOfRegions` is a maximum limit, right? Maybe call it `limit`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by gokulavasan <gi...@git.apache.org>.
Github user gokulavasan commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99964049
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -129,17 +168,19 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
Result next;
while ((next = scanner.next()) != null) {
byte[] region = getRegionFromKey(next.getRow());
- if (regions.contains(region)) {
- byte[] timeBytes = next.getValue(FAMILY, PRUNE_UPPER_BOUND_COL);
- if (timeBytes != null) {
- long pruneUpperBoundRegion = Bytes.toLong(timeBytes);
- resultMap.put(region, pruneUpperBoundRegion);
+ if (regions == null || regions.contains(region)) {
+ Cell cell = next.getColumnLatestCell(FAMILY, PRUNE_UPPER_BOUND_COL);
+ if (cell != null) {
+ byte[] pruneUpperBoundBytes = CellUtil.cloneValue(cell);
+ long timestamp = cell.getTimestamp();
--- End diff --
PruneRecordTime ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750520
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
+ final Connection connection = ConnectionFactory.createConnection(conf);
+ dataJanitorState = new DataJanitorState(new DataJanitorState.TableSupplier() {
+ @Override
+ public Table get() throws IOException {
+ return connection.getTable(TableName.valueOf(
+ conf.get(TxConstants.TransactionPruning.PRUNE_STATE_TABLE,
+ TxConstants.TransactionPruning.DEFAULT_PRUNE_STATE_TABLE)));
+ }
+ });
+ }
+
+ /**
+ * Return a list of RegionPruneInfo. These regions are the ones that have the lowest prune upper bounds.
+ * If -1 is passed in, all the regions and their prune upper bound will be returned.
+ *
+ * @param numRegions number of regions
+ * @return Map of region name and its prune upper bound
+ */
+ public Queue<RegionPruneInfo> getIdleRegions(Integer numRegions) throws IOException {
+ List<RegionPruneInfo> regionPruneInfos = dataJanitorState.getPruneInfoForRegions(null);
+ if (regionPruneInfos.isEmpty()) {
+ return new LinkedList<>();
+ }
+
+ if (numRegions < 0) {
+ numRegions = regionPruneInfos.size();
+ }
+
+ Queue<RegionPruneInfo> lowestPrunes = MinMaxPriorityQueue.orderedBy(new Comparator<RegionPruneInfo>() {
+ @Override
+ public int compare(RegionPruneInfo o1, RegionPruneInfo o2) {
+ return (int) (o1.getPruneUpperBound() - o2.getPruneUpperBound());
+ }
+ }).maximumSize(numRegions).create();
+
+ for (RegionPruneInfo pruneInfo : regionPruneInfos) {
+ lowestPrunes.add(pruneInfo);
+ }
+ return lowestPrunes;
+ }
+
+ /**
+ * Return the prune upper bound value of a given region. If no prune upper bound has been written for this region yet,
+ * it will return a null.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} of the region
+ * @throws IOException if there are any errors while trying to fetch the {@link RegionPruneInfo}
+ */
+ @Nullable
+ public RegionPruneInfo getRegionPruneInfo(String regionId) throws IOException {
+ return dataJanitorState.getPruneInfoForRegion(Bytes.toBytesBinary(regionId));
+ }
+
+ /**
+ *
+ * @param time Given a time, provide the {@link TimeRegions} at or before that time
+ * @return time regions or null if no time regions are present at or before the given time
+ * @throws IOException if there are any errors while trying to fetch the {@link TimeRegions}
+ */
+ @Nullable
+ public String getRegionsOnOrBeforeTime(Long time) throws IOException {
+ TimeRegions timeRegions = dataJanitorState.getRegionsOnOrBeforeTime(time);
+ if (timeRegions == null) {
+ return null;
+ }
+ return timeRegions.toString();
+ }
+
+ private void printUsage(PrintWriter pw) {
+ pw.println("Usage : org.apache.tephra.hbase.txprune.InvalidListPruning <command> <parameter>");
+ pw.println("Available commands, corresponding parameters are:");
+ pw.println("****************************************************");
+ pw.println("timeregion ts");
+ pw.println("Desc: Prints out a time region at time 'ts' (in milliseconds) or the latest before time 'ts'.");
+ pw.println("idleregions numOfRegions");
+ pw.println("Desc: Prints out 'numOfRegions' number of regions which has the lowest prune Upper bounds.");
--- End diff --
Also add `-1` as limit gives all
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99750227
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/InvalidListPruningDebug.java ---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tephra.hbase.txprune;
+
+import com.google.common.collect.MinMaxPriorityQueue;
+import com.google.gson.Gson;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.tephra.TxConstants;
+import org.apache.tephra.txprune.RegionPruneInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.util.Comparator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+import javax.annotation.Nullable;
+
+/**
+ * Invalid List Pruning Debug Tool.
+ */
+public class InvalidListPruningDebug {
+ private static final Logger LOG = LoggerFactory.getLogger(InvalidListPruningDebug.class);
+ private static final Gson GSON = new Gson();
+ private DataJanitorState dataJanitorState;
+
+ /**
+ * Initialize the Invalid List Debug Tool.
+ * @param conf {@link Configuration}
+ * @throws IOException
+ */
+ public void initialize(final Configuration conf) throws IOException {
+ LOG.info("InvalidListPruningDebugMain : initialize method called");
--- End diff --
This can be changed to debug log
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99760787
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -129,17 +168,19 @@ public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
Result next;
while ((next = scanner.next()) != null) {
byte[] region = getRegionFromKey(next.getRow());
- if (regions.contains(region)) {
- byte[] timeBytes = next.getValue(FAMILY, PRUNE_UPPER_BOUND_COL);
- if (timeBytes != null) {
- long pruneUpperBoundRegion = Bytes.toLong(timeBytes);
- resultMap.put(region, pruneUpperBoundRegion);
+ if (regions == null || regions.contains(region)) {
+ Cell cell = next.getColumnLatestCell(FAMILY, PRUNE_UPPER_BOUND_COL);
+ if (cell != null) {
+ byte[] pruneUpperBoundBytes = CellUtil.cloneValue(cell);
+ long timestamp = cell.getTimestamp();
--- End diff --
Also it may be okay to treat this as the compaction timestamp for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tephra/pull/31
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tephra pull request #31: (TEPHRA-214) Tool to debug the state and ...
Posted by poornachandra <gi...@git.apache.org>.
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-tephra/pull/31#discussion_r99759970
--- Diff: tephra-hbase-compat-1.1-base/src/main/java/org/apache/tephra/hbase/txprune/DataJanitorState.java ---
@@ -102,11 +107,29 @@ public void savePruneUpperBoundForRegion(byte[] regionId, long pruneUpperBound)
* @throws IOException when not able to read the data from HBase
*/
public long getPruneUpperBoundForRegion(byte[] regionId) throws IOException {
+ RegionPruneInfo regionPruneInfo = getPruneInfoForRegion(regionId);
+ return (regionPruneInfo == null) ? -1 : regionPruneInfo.getCompactionTimestamp();
+ }
+
+ /**
+ * Get the latest {@link RegionPruneInfo} for a given region.
+ *
+ * @param regionId region id
+ * @return {@link RegionPruneInfo} for the region
+ * @throws IOException when not able to read the data from HBase
+ */
+ public RegionPruneInfo getPruneInfoForRegion(byte[] regionId) throws IOException {
--- End diff --
This method needs to be annotated as `@Nullable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---