You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/01/24 02:54:32 UTC
[GitHub] [phoenix] kadirozde opened a new pull request #691: PHOENIX-5694
Add MR job counters for IndexTool inline verification
kadirozde opened a new pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370894846
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -298,69 +655,71 @@ private boolean checkIndexRow(final byte[] indexRowKey, final Put put) throws IO
}
private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, long dataRowTs, long indexRowTs,
- String errorMsg) {
+ String errorMsg) throws IOException {
logToIndexToolOutputTable(dataRowKey, indexRowKey, dataRowTs, indexRowTs,
errorMsg, null, null);
}
private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, long dataRowTs, long indexRowTs,
- String errorMsg, byte[] expectedValue, byte[] actualValue) {
+ String errorMsg, byte[] expectedValue, byte[] actualValue) throws IOException {
final byte[] E_VALUE_PREFIX_BYTES = Bytes.toBytes(" E:");
final byte[] A_VALUE_PREFIX_BYTES = Bytes.toBytes(" A:");
final int PREFIX_LENGTH = 3;
final int TOTAL_PREFIX_LENGTH = 6;
+ final byte[] PHASE_BEFORE_VALUE = Bytes.toBytes("BEFORE");
+ final byte[] PHASE_AFTER_VALUE = Bytes.toBytes("AFTER");
+ long scanMaxTs = scan.getTimeRange().getMax();
+ byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
+ byte[] rowKey;
+ if (dataRowKey != null) {
Review comment:
Again, good to document what the PK schema is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370948407
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -298,69 +655,71 @@ private boolean checkIndexRow(final byte[] indexRowKey, final Put put) throws IO
}
private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, long dataRowTs, long indexRowTs,
- String errorMsg) {
+ String errorMsg) throws IOException {
logToIndexToolOutputTable(dataRowKey, indexRowKey, dataRowTs, indexRowTs,
errorMsg, null, null);
}
private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, long dataRowTs, long indexRowTs,
- String errorMsg, byte[] expectedValue, byte[] actualValue) {
+ String errorMsg, byte[] expectedValue, byte[] actualValue) throws IOException {
final byte[] E_VALUE_PREFIX_BYTES = Bytes.toBytes(" E:");
final byte[] A_VALUE_PREFIX_BYTES = Bytes.toBytes(" A:");
final int PREFIX_LENGTH = 3;
final int TOTAL_PREFIX_LENGTH = 6;
+ final byte[] PHASE_BEFORE_VALUE = Bytes.toBytes("BEFORE");
+ final byte[] PHASE_AFTER_VALUE = Bytes.toBytes("AFTER");
+ long scanMaxTs = scan.getTimeRange().getMax();
+ byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
+ byte[] rowKey;
+ if (dataRowKey != null) {
Review comment:
I will do that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370945765
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexToolIT.java
##########
@@ -521,17 +552,14 @@ public void testIndexToolVerifyAfterOption() throws Exception {
assertTrue(Bytes.compareTo(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength(),
expectedValueBytes, 0, expectedValueBytes.length) == 0);
IndexRegionObserver.setIgnoreIndexRebuildForTesting(false);
- Admin admin = conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin();
- TableName indexToolOutputTable = TableName.valueOf(IndexTool.OUTPUT_TABLE_NAME_BYTES);
- admin.disableTable(indexToolOutputTable);
- admin.deleteTable(indexToolOutputTable);
+ dropIndexToolTables(conn);
Review comment:
Some tests do nothing for some parameters and these tables will not be created for them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370946517
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
Review comment:
Good suggestion. I will also eliminate isXXXX(Cell cell) methods altogether.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370893828
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
Review comment:
if this method knew what phase it was in you could eliminate half the comparisons, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370947778
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
+ addScannedDataRowCount(getValue(cell));
+ } else if (isRebuiltIndexRowCount(cell)) {
+ addRebuiltIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildValidIndexRowCount(cell)) {
+ addBeforeRebuildValidIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildExpiredIndexRowCount(cell)) {
+ addBeforeRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildMissingIndexRowCount(cell)) {
+ addBeforeRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildInvalidIndexRowCount(cell)) {
+ addBeforeRebuildInvalidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildValidIndexRowCount(cell)) {
+ addAfterRebuildValidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildExpiredIndexRowCount(cell)) {
+ addAfterRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildMissingIndexRowCount(cell)) {
+ addAfterRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildInvalidIndexRowCount(cell)) {
+ addAfterRebuildInvalidIndexRowCount(getValue(cell));
+ }
+ }
+
+ public static byte[] calculateTheClosestNextRowKeyForPrefix(byte[] rowKeyPrefix) {
+ // Essentially we are treating it like an 'unsigned very very long' and doing +1 manually.
+ // Search for the place where the trailing 0xFFs start
+ int offset = rowKeyPrefix.length;
+ while (offset > 0) {
+ if (rowKeyPrefix[offset - 1] != (byte) 0xFF) {
+ break;
+ }
+ offset--;
+ }
+ if (offset == 0) {
+ // We got an 0xFFFF... (only FFs) stopRow value which is
+ // the last possible prefix before the end of the table.
+ // So set it to stop at the 'end of the table'
+ return HConstants.EMPTY_END_ROW;
+ }
+ // Copy the right length of the original
+ byte[] newStopRow = Arrays.copyOfRange(rowKeyPrefix, 0, offset);
+ // And increment the last one
+ newStopRow[newStopRow.length - 1]++;
+ return newStopRow;
+ }
+
+ public static VerificationResult getVerificationResult(Table hTable, long ts)
+ throws IOException {
+ VerificationResult verificationResult = new VerificationResult();
+ byte[] startRowKey = Bytes.toBytes(Long.toString(ts));
+ byte[] stopRowKey = calculateTheClosestNextRowKeyForPrefix(startRowKey);
+ Scan scan = new Scan();
+ scan.setStartRow(startRowKey);
+ scan.setStopRow(stopRowKey);
+ ResultScanner scanner = hTable.getScanner(scan);
+ for (Result result = scanner.next(); result != null; result = scanner.next()) {
+ for (Cell cell : result.rawCells()) {
+ verificationResult.update(cell);
+ }
+ }
+ return verificationResult;
+ }
+
+ public boolean isVerificationFailed(IndexTool.IndexVerifyType verifyType) {
+ if (verifyType == IndexTool.IndexVerifyType.BEFORE || verifyType == IndexTool.IndexVerifyType.NONE) {
Review comment:
BEFORE means build only invalid or missing rows. Here the verification is not for verifying the rows that have been rebuilt. Instead, it is for identifying the rows to be rebuilt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370894419
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -187,13 +509,56 @@ public RegionInfo getRegionInfo() {
@Override
public boolean isFilterDone() { return hasMore; }
+ private void logToIndexToolResultTable() throws IOException {
+ long scanMaxTs = scan.getTimeRange().getMax();
+ byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
+ byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
+ byte[] rowKey = new byte[keyPrefix.length + regionName.length];
+ Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
Review comment:
Comment here that PK is scanMaxTs + regionName would be helpful. Or alternately, extract key building into a helper method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370897416
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexTool.java
##########
@@ -641,18 +668,27 @@ public Job getJob() {
return job;
}
- private void createIndexToolOutputTable(Connection connection) throws Exception {
+ private void createIndexToolTables(Connection connection) throws Exception {
ConnectionQueryServices queryServices = connection.unwrap(PhoenixConnection.class).getQueryServices();
Admin admin = queryServices.getAdmin();
- if (admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
- return;
+ if (!admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
Review comment:
will this work on a cluster with namespaces enabled?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r371044146
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -680,20 +609,21 @@ private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, lo
rowKey = new byte[keyPrefix.length];
Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
}
+ //TODO Write timestamps as strings instead of longs
Review comment:
Do you still intend to do this TODO and change the longs to strings? If so, why not do it now, and if not, why not get rid of the TODO? :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370794080
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexTool.java
##########
@@ -641,18 +668,27 @@ public Job getJob() {
return job;
}
- private void createIndexToolOutputTable(Connection connection) throws Exception {
+ private void createIndexToolTables(Connection connection) throws Exception {
ConnectionQueryServices queryServices = connection.unwrap(PhoenixConnection.class).getQueryServices();
Admin admin = queryServices.getAdmin();
- if (admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
- return;
+ if (!admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
+ HTableDescriptor tableDescriptor = new
+ HTableDescriptor(TableName.valueOf(OUTPUT_TABLE_NAME));
+ tableDescriptor.setValue("DISABLE_TABLE_SOR", "true");
+ tableDescriptor.setValue(HColumnDescriptor.TTL, String.valueOf(MetaDataProtocol.DEFAULT_LOG_TTL));
+ HColumnDescriptor columnDescriptor = new HColumnDescriptor(OUTPUT_TABLE_COLUMN_FAMILY);
+ tableDescriptor.addFamily(columnDescriptor);
+ admin.createTable(tableDescriptor);
+ }
+ if (!admin.tableExists(TableName.valueOf(RESULT_TABLE_NAME))) {
+ HTableDescriptor tableDescriptor = new
+ HTableDescriptor(TableName.valueOf(RESULT_TABLE_NAME));
+ tableDescriptor.setValue("DISABLE_TABLE_SOR", "true");
+ tableDescriptor.setValue(HColumnDescriptor.TTL, String.valueOf(MetaDataProtocol.DEFAULT_LOG_TTL));
+ HColumnDescriptor columnDescriptor = new HColumnDescriptor(RESULT_TABLE_COLUMN_FAMILY);
+ tableDescriptor.addFamily(columnDescriptor);
+ admin.createTable(tableDescriptor);
Review comment:
I will remove tableDescriptor.setValue("DISABLE_TABLE_SOR", "true") lines above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370460133
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/IndexToolIT.java
##########
@@ -521,17 +552,14 @@ public void testIndexToolVerifyAfterOption() throws Exception {
assertTrue(Bytes.compareTo(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength(),
expectedValueBytes, 0, expectedValueBytes.length) == 0);
IndexRegionObserver.setIgnoreIndexRebuildForTesting(false);
- Admin admin = conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin();
- TableName indexToolOutputTable = TableName.valueOf(IndexTool.OUTPUT_TABLE_NAME_BYTES);
- admin.disableTable(indexToolOutputTable);
- admin.deleteTable(indexToolOutputTable);
+ dropIndexToolTables(conn);
Review comment:
should these dropIndexToolTables be in an After method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370897715
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexImportDirectReducer.java
##########
@@ -40,15 +43,79 @@
import static org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
+import static org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getIndexVerifyType;
+
/**
* Reducer class that does only one task and that is to update the index state of the table.
*/
public class PhoenixIndexImportDirectReducer extends
Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable> {
-
+ private boolean calledOnce = false;
private static final Logger LOGGER =
LoggerFactory.getLogger(PhoenixIndexImportDirectReducer.class);
+ private void updateCounters(IndexTool.IndexVerifyType verifyType,
+ Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable>.Context context)
+ throws IOException {
+ Configuration configuration = context.getConfiguration();
+ try (final Connection connection = ConnectionUtil.getInputConnection(configuration)) {
+ long ts = Long.valueOf(configuration.get(PhoenixConfigurationUtil.CURRENT_SCN_VALUE));
+ Table hTable = connection.unwrap(PhoenixConnection.class).getQueryServices()
+ .getTable(IndexTool.RESULT_TABLE_NAME_BYTES);
+ IndexRebuildRegionScanner.VerificationResult verificationResult =
+ IndexRebuildRegionScanner.VerificationResult.getVerificationResult(hTable, ts);
+ context.getCounter(PhoenixIndexToolJobCounters.SCANNED_DATA_ROW_COUNT).
+ setValue(verificationResult.getScannedDataRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.REBUILT_INDEX_ROW_COUNT).
+ setValue(verificationResult.getRebuiltIndexRowCount());
+ if (verifyType == IndexTool.IndexVerifyType.ONLY || verifyType == IndexTool.IndexVerifyType.BEFORE ||
+ verifyType == IndexTool.IndexVerifyType.BOTH) {
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildValidIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildExpiredIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildMissingIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildInvalidIndexRowCount());
+ }
+ if (verifyType == IndexTool.IndexVerifyType.BOTH || verifyType == IndexTool.IndexVerifyType.AFTER) {
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_VALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildValidIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildExpiredIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildMissingIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildInvalidIndexRowCount());
+ }
+ if (verificationResult.isVerificationFailed(verifyType)) {
+ throw new IOException("Index verification failed! " + verificationResult);
+ }
+ } catch (Exception e) {
+ throw new IOException("Fail to get index verification result", e);
+ }
+ }
+
+ @Override
+ protected void reduce(ImmutableBytesWritable arg0, Iterable<IntWritable> arg1,
+ Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable>.Context context)
+ throws IOException, InterruptedException
+
+ {
+ synchronized (this) {
Review comment:
can use an AtomicBoolean instead of a synchronized block.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370949846
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/PhoenixIndexImportDirectReducer.java
##########
@@ -40,15 +43,79 @@
import static org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
+import static org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getIndexVerifyType;
+
/**
* Reducer class that does only one task and that is to update the index state of the table.
*/
public class PhoenixIndexImportDirectReducer extends
Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable> {
-
+ private boolean calledOnce = false;
private static final Logger LOGGER =
LoggerFactory.getLogger(PhoenixIndexImportDirectReducer.class);
+ private void updateCounters(IndexTool.IndexVerifyType verifyType,
+ Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable>.Context context)
+ throws IOException {
+ Configuration configuration = context.getConfiguration();
+ try (final Connection connection = ConnectionUtil.getInputConnection(configuration)) {
+ long ts = Long.valueOf(configuration.get(PhoenixConfigurationUtil.CURRENT_SCN_VALUE));
+ Table hTable = connection.unwrap(PhoenixConnection.class).getQueryServices()
+ .getTable(IndexTool.RESULT_TABLE_NAME_BYTES);
+ IndexRebuildRegionScanner.VerificationResult verificationResult =
+ IndexRebuildRegionScanner.VerificationResult.getVerificationResult(hTable, ts);
+ context.getCounter(PhoenixIndexToolJobCounters.SCANNED_DATA_ROW_COUNT).
+ setValue(verificationResult.getScannedDataRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.REBUILT_INDEX_ROW_COUNT).
+ setValue(verificationResult.getRebuiltIndexRowCount());
+ if (verifyType == IndexTool.IndexVerifyType.ONLY || verifyType == IndexTool.IndexVerifyType.BEFORE ||
+ verifyType == IndexTool.IndexVerifyType.BOTH) {
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildValidIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildExpiredIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildMissingIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getBeforeRebuildInvalidIndexRowCount());
+ }
+ if (verifyType == IndexTool.IndexVerifyType.BOTH || verifyType == IndexTool.IndexVerifyType.AFTER) {
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_VALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildValidIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildExpiredIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildMissingIndexRowCount());
+ context.getCounter(PhoenixIndexToolJobCounters.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT).
+ setValue(verificationResult.getAfterRebuildInvalidIndexRowCount());
+ }
+ if (verificationResult.isVerificationFailed(verifyType)) {
+ throw new IOException("Index verification failed! " + verificationResult);
+ }
+ } catch (Exception e) {
+ throw new IOException("Fail to get index verification result", e);
+ }
+ }
+
+ @Override
+ protected void reduce(ImmutableBytesWritable arg0, Iterable<IntWritable> arg1,
+ Reducer<ImmutableBytesWritable, IntWritable, NullWritable, NullWritable>.Context context)
+ throws IOException, InterruptedException
+
+ {
+ synchronized (this) {
Review comment:
I will use it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370947555
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
+ addScannedDataRowCount(getValue(cell));
+ } else if (isRebuiltIndexRowCount(cell)) {
+ addRebuiltIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildValidIndexRowCount(cell)) {
+ addBeforeRebuildValidIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildExpiredIndexRowCount(cell)) {
+ addBeforeRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildMissingIndexRowCount(cell)) {
+ addBeforeRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildInvalidIndexRowCount(cell)) {
+ addBeforeRebuildInvalidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildValidIndexRowCount(cell)) {
+ addAfterRebuildValidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildExpiredIndexRowCount(cell)) {
+ addAfterRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildMissingIndexRowCount(cell)) {
+ addAfterRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildInvalidIndexRowCount(cell)) {
+ addAfterRebuildInvalidIndexRowCount(getValue(cell));
+ }
+ }
+
+ public static byte[] calculateTheClosestNextRowKeyForPrefix(byte[] rowKeyPrefix) {
+ // Essentially we are treating it like an 'unsigned very very long' and doing +1 manually.
+ // Search for the place where the trailing 0xFFs start
+ int offset = rowKeyPrefix.length;
Review comment:
I got the calculateTheClosestNextRowKeyForPrefix method from HBase 3.0. When we start using HBase 3.0, we can remove this local copy. It seems Bytes.incrementBytes does something similar but not exactly the same, I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r371059430
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -680,20 +609,21 @@ private void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, lo
rowKey = new byte[keyPrefix.length];
Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
}
+ //TODO Write timestamps as strings instead of longs
Review comment:
I have done with it but forgot to remove the comment. I will remove it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370948321
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -187,13 +509,56 @@ public RegionInfo getRegionInfo() {
@Override
public boolean isFilterDone() { return hasMore; }
+ private void logToIndexToolResultTable() throws IOException {
+ long scanMaxTs = scan.getTimeRange().getMax();
+ byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
+ byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
+ byte[] rowKey = new byte[keyPrefix.length + regionName.length];
+ Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
Review comment:
I will add the comment you suggested
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370949623
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/index/IndexTool.java
##########
@@ -641,18 +668,27 @@ public Job getJob() {
return job;
}
- private void createIndexToolOutputTable(Connection connection) throws Exception {
+ private void createIndexToolTables(Connection connection) throws Exception {
ConnectionQueryServices queryServices = connection.unwrap(PhoenixConnection.class).getQueryServices();
Admin admin = queryServices.getAdmin();
- if (admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
- return;
+ if (!admin.tableExists(TableName.valueOf(OUTPUT_TABLE_NAME))) {
Review comment:
It should work. Tables can be created without specifying a namespace. Then they are created within the default namespace and the default namespace is not specified in commands/operations as far as I understand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370465938
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
+ addScannedDataRowCount(getValue(cell));
+ } else if (isRebuiltIndexRowCount(cell)) {
+ addRebuiltIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildValidIndexRowCount(cell)) {
+ addBeforeRebuildValidIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildExpiredIndexRowCount(cell)) {
+ addBeforeRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildMissingIndexRowCount(cell)) {
+ addBeforeRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildInvalidIndexRowCount(cell)) {
+ addBeforeRebuildInvalidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildValidIndexRowCount(cell)) {
+ addAfterRebuildValidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildExpiredIndexRowCount(cell)) {
+ addAfterRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildMissingIndexRowCount(cell)) {
+ addAfterRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildInvalidIndexRowCount(cell)) {
+ addAfterRebuildInvalidIndexRowCount(getValue(cell));
+ }
+ }
+
+ public static byte[] calculateTheClosestNextRowKeyForPrefix(byte[] rowKeyPrefix) {
+ // Essentially we are treating it like an 'unsigned very very long' and doing +1 manually.
+ // Search for the place where the trailing 0xFFs start
+ int offset = rowKeyPrefix.length;
Review comment:
Does Bytes.incrementBytes accomplish the same thing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde closed pull request #691: PHOENIX-5694 Add MR
job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde closed pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370461543
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
Review comment:
for all of these isXXXX(Cell cell) methods, see CellUtil.matchingColumn() to greatly simplify. For example, this becomes "return CellUtil.matchingColumn(cell, IndexTool.SCANNED_DATA_ROW_COUNT_BYTES)"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370466507
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
+ addScannedDataRowCount(getValue(cell));
+ } else if (isRebuiltIndexRowCount(cell)) {
+ addRebuiltIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildValidIndexRowCount(cell)) {
+ addBeforeRebuildValidIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildExpiredIndexRowCount(cell)) {
+ addBeforeRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildMissingIndexRowCount(cell)) {
+ addBeforeRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isBeforeRebuildInvalidIndexRowCount(cell)) {
+ addBeforeRebuildInvalidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildValidIndexRowCount(cell)) {
+ addAfterRebuildValidIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildExpiredIndexRowCount(cell)) {
+ addAfterRebuildExpiredIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildMissingIndexRowCount(cell)) {
+ addAfterRebuildMissingIndexRowCount(getValue(cell));
+ } else if (isAfterRebuildInvalidIndexRowCount(cell)) {
+ addAfterRebuildInvalidIndexRowCount(getValue(cell));
+ }
+ }
+
+ public static byte[] calculateTheClosestNextRowKeyForPrefix(byte[] rowKeyPrefix) {
+ // Essentially we are treating it like an 'unsigned very very long' and doing +1 manually.
+ // Search for the place where the trailing 0xFFs start
+ int offset = rowKeyPrefix.length;
+ while (offset > 0) {
+ if (rowKeyPrefix[offset - 1] != (byte) 0xFF) {
+ break;
+ }
+ offset--;
+ }
+ if (offset == 0) {
+ // We got an 0xFFFF... (only FFs) stopRow value which is
+ // the last possible prefix before the end of the table.
+ // So set it to stop at the 'end of the table'
+ return HConstants.EMPTY_END_ROW;
+ }
+ // Copy the right length of the original
+ byte[] newStopRow = Arrays.copyOfRange(rowKeyPrefix, 0, offset);
+ // And increment the last one
+ newStopRow[newStopRow.length - 1]++;
+ return newStopRow;
+ }
+
+ public static VerificationResult getVerificationResult(Table hTable, long ts)
+ throws IOException {
+ VerificationResult verificationResult = new VerificationResult();
+ byte[] startRowKey = Bytes.toBytes(Long.toString(ts));
+ byte[] stopRowKey = calculateTheClosestNextRowKeyForPrefix(startRowKey);
+ Scan scan = new Scan();
+ scan.setStartRow(startRowKey);
+ scan.setStopRow(stopRowKey);
+ ResultScanner scanner = hTable.getScanner(scan);
+ for (Result result = scanner.next(); result != null; result = scanner.next()) {
+ for (Cell cell : result.rawCells()) {
+ verificationResult.update(cell);
+ }
+ }
+ return verificationResult;
+ }
+
+ public boolean isVerificationFailed(IndexTool.IndexVerifyType verifyType) {
+ if (verifyType == IndexTool.IndexVerifyType.BEFORE || verifyType == IndexTool.IndexVerifyType.NONE) {
Review comment:
why can verification not fail if it's BEFORE?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [phoenix] kadirozde commented on a change in pull request #691:
PHOENIX-5694 Add MR job counters for IndexTool inline verification
Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #691: PHOENIX-5694 Add MR job counters for IndexTool inline verification
URL: https://github.com/apache/phoenix/pull/691#discussion_r370948120
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
##########
@@ -89,6 +94,319 @@
import com.google.common.collect.Sets;
public class IndexRebuildRegionScanner extends BaseRegionScanner {
+
+ public static class VerificationResult {
+ public static class PhaseResult {
+ private long validIndexRowCount = 0;
+ private long expiredIndexRowCount = 0;
+ private long missingIndexRowCount = 0;
+ private long invalidIndexRowCount = 0;
+
+ public void add(PhaseResult phaseResult) {
+ validIndexRowCount += phaseResult.validIndexRowCount;
+ expiredIndexRowCount += phaseResult.expiredIndexRowCount;
+ missingIndexRowCount += phaseResult.missingIndexRowCount;
+ invalidIndexRowCount += phaseResult.invalidIndexRowCount;
+ }
+
+ public long getTotalCount() {
+ return validIndexRowCount + expiredIndexRowCount + missingIndexRowCount + invalidIndexRowCount;
+ }
+
+ @Override
+ public String toString() {
+ return "PhaseResult{" +
+ "validIndexRowCount=" + validIndexRowCount +
+ ", expiredIndexRowCount=" + expiredIndexRowCount +
+ ", missingIndexRowCount=" + missingIndexRowCount +
+ ", invalidIndexRowCount=" + invalidIndexRowCount +
+ '}';
+ }
+ }
+
+ private long scannedDataRowCount = 0;
+ private long rebuiltIndexRowCount = 0;
+ private PhaseResult before = new PhaseResult();
+ private PhaseResult after = new PhaseResult();
+
+ @Override
+ public String toString() {
+ return "VerificationResult{" +
+ "scannedDataRowCount=" + scannedDataRowCount +
+ ", rebuiltIndexRowCount=" + rebuiltIndexRowCount +
+ ", before=" + before +
+ ", after=" + after +
+ '}';
+ }
+
+ public long getScannedDataRowCount() {
+ return scannedDataRowCount;
+ }
+
+ public long getRebuiltIndexRowCount() {
+ return rebuiltIndexRowCount;
+ }
+
+ public long getBeforeRebuildValidIndexRowCount() {
+ return before.validIndexRowCount;
+ }
+
+ public long getBeforeRebuildExpiredIndexRowCount() {
+ return before.expiredIndexRowCount;
+ }
+
+ public long getBeforeRebuildInvalidIndexRowCount() {
+ return before.invalidIndexRowCount;
+ }
+
+ public long getBeforeRebuildMissingIndexRowCount() {
+ return before.missingIndexRowCount;
+ }
+
+ public long getAfterRebuildValidIndexRowCount() {
+ return after.validIndexRowCount;
+ }
+
+ public long getAfterRebuildExpiredIndexRowCount() {
+ return after.expiredIndexRowCount;
+ }
+
+ public long getAfterRebuildInvalidIndexRowCount() {
+ return after.invalidIndexRowCount;
+ }
+
+ public long getAfterRebuildMissingIndexRowCount() {
+ return after.missingIndexRowCount;
+ }
+
+ private void addScannedDataRowCount(long count) {
+ this.scannedDataRowCount += count;
+ }
+
+ private void addRebuiltIndexRowCount(long count) {
+ this.rebuiltIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildValidIndexRowCount(long count) {
+ before.validIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildExpiredIndexRowCount(long count) {
+ before.expiredIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildMissingIndexRowCount(long count) {
+ before.missingIndexRowCount += count;
+ }
+
+ private void addBeforeRebuildInvalidIndexRowCount(long count) {
+ before.invalidIndexRowCount += count;
+ }
+
+ private void addAfterRebuildValidIndexRowCount(long count) {
+ after.validIndexRowCount += count;
+ }
+
+ private void addAfterRebuildExpiredIndexRowCount(long count) {
+ after.expiredIndexRowCount += count;
+ }
+
+ private void addAfterRebuildMissingIndexRowCount(long count) {
+ after.missingIndexRowCount += count;
+ }
+
+ private void addAfterRebuildInvalidIndexRowCount(long count) {
+ after.invalidIndexRowCount += count;
+ }
+
+ private static boolean isScannedDataRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES, 0,
+ IndexTool.SCANNED_DATA_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isRebuiltIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.REBUILT_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isBeforeRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.BEFORE_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildValidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_VALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildExpiredIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_EXPIRED_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildMissingIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_MISSING_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private static boolean isAfterRebuildInvalidIndexRowCount(Cell cell) {
+ if (Bytes.compareTo(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength(),
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES, 0,
+ IndexTool.AFTER_REBUILD_INVALID_INDEX_ROW_COUNT_BYTES.length) == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ private long getValue(Cell cell) {
+ return Long.parseLong(Bytes.toString(cell.getValueArray(),
+ cell.getValueOffset(), cell.getValueLength()));
+ }
+
+ private void update(Cell cell) {
+ if (isScannedDataRowCount(cell)) {
Review comment:
This method is called only once and by the reducer. It is not called by the mapper which implements the phases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services