You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/03/18 22:08:52 UTC

[GitHub] [phoenix] kadirozde opened a new pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

kadirozde opened a new pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737
 
 
   …neous rebuilds

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395449605
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + regionName.length + ROW_KEY_SEPARATOR_BYTE.length +
+                scan.getStartRow().length + ROW_KEY_SEPARATOR_BYTE.length + scan.getStopRow().length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, indexHTable.getName().toBytes(), 0, indexHTable.getName().toBytes().length);
+        targetOffset += indexHTable.getName().toBytes().length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, regionName, 0, regionName.length);
+        targetOffset += regionName.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, scan.getStartRow(), 0, scan.getStartRow().length);
+        targetOffset += scan.getStartRow().length;
 
 Review comment:
   @gjacoby126, I am not sure what you are really asking. I did not say these separators are for programs. I said they were just there to help humans. The previous version did not have any separators. There were no requirement in this jira to add such separators. I added them based on Priyank's suggestion as he claimed it would improve debugging. If there is a requirement to parse the row key to obtain subfields then we need to have a separate jira to implement that. Please also note that the row key is constructed in a way to help debugging. For example, the timestamp values are written as variable strings instead of fixed-size long values. We need to redesign the row key and maybe to have two versions of it, one for human and the other for programs.  That deserves a separate jira.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#issuecomment-602838272
 
 
   > > @kadirozde - I offered (and am still offering) to split out the code into a separate class and write read / field parsing logic in a separate JIRA, but I believe @swaroopak 's request for a test to cover the write logic you've changed is still needed for this JIRA.
   > 
   > @gjacoby126, the change is covered by many existing tests already. You cannot get verification counters correct if the key is not constructed properly. As I said before, this JIRA does not offer anything for parsing or extracting subfields within a row key.
   
   
   
   > > @kadirozde - I offered (and am still offering) to split out the code into a separate class and write read / field parsing logic in a separate JIRA, but I believe @swaroopak 's request for a test to cover the write logic you've changed is still needed for this JIRA.
   > 
   > @gjacoby126, the change is covered by many existing tests already. You cannot get verification counters correct if the key is not constructed properly. As I said before, this JIRA does not offer anything for parsing or extracting subfields within a row key.
   
   Okay. I have added testing for row keys.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#issuecomment-602150612
 
 
   > A small test to cover the changed part of the code would be highly appreciated. Thanks.
   
   @swaroopak, @gjacoby126 has offered to extract the code for index tool tables out and write unit tests for it. As part of that effort, the unit tests for index tool table row keys can be added.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395322329
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
 
 Review comment:
   Extracting the index tool result and index tool output writing code into separate classes (with read methods!) would facilitate testing -- I agree with @swaroopak that this needs some tests, I'd say probably for both insert and read. Adding read methods would also make life easier for utilities, whether in Phoenix or outside, to be able to programmatically parse the result and output tables. Otherwise a whole lot of boilerplate byte parsing has to be rewritten each time. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395388584
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -412,30 +426,31 @@ public void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, lon
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] rowKey;
-        // The row key for the output table is the max timestamp of the scan + data row key
-        if (dataRowKey != null) {
-            rowKey = new byte[keyPrefix.length + dataRowKey.length];
-            Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-            Bytes.putBytes(rowKey, keyPrefix.length, dataRowKey, 0, dataRowKey.length);
-        } else {
-            rowKey = new byte[keyPrefix.length];
-            Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        }
+        int targetOffset = 0;
+        // The row key for the output table : timestamp | index table name | data row key
+        rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + dataRowKey.length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
 
 Review comment:
   Will do that

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395319356
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + regionName.length + ROW_KEY_SEPARATOR_BYTE.length +
+                scan.getStartRow().length + ROW_KEY_SEPARATOR_BYTE.length + scan.getStopRow().length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, indexHTable.getName().toBytes(), 0, indexHTable.getName().toBytes().length);
+        targetOffset += indexHTable.getName().toBytes().length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, regionName, 0, regionName.length);
+        targetOffset += regionName.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, scan.getStartRow(), 0, scan.getStartRow().length);
+        targetOffset += scan.getStartRow().length;
 
 Review comment:
   What happens if the Scan start row contains the bytes for "|"? Does this need to be length encoded instead of using separators? Or perhaps escaped somehow?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395387700
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + regionName.length + ROW_KEY_SEPARATOR_BYTE.length +
+                scan.getStartRow().length + ROW_KEY_SEPARATOR_BYTE.length + scan.getStopRow().length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, indexHTable.getName().toBytes(), 0, indexHTable.getName().toBytes().length);
+        targetOffset += indexHTable.getName().toBytes().length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, regionName, 0, regionName.length);
+        targetOffset += regionName.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, scan.getStartRow(), 0, scan.getStartRow().length);
+        targetOffset += scan.getStartRow().length;
 
 Review comment:
   Assume that a non-human has to parse this key into fields, and that this is not just for use manually in HBase shell. What happens if the Scan start row contains the bytes for "|"? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#issuecomment-602666121
 
 
   @kadirozde - I offered (and am still offering) to split out the code into a separate class and write read / field parsing logic in a separate JIRA, but I believe @swaroopak 's request for a test to cover the write logic you've changed is still needed for this JIRA. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395446572
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
 
 Review comment:
   Please feel free to open a JIRA to extract  index tool output writing code into separate classes and  write a unit test for them.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395386410
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + regionName.length + ROW_KEY_SEPARATOR_BYTE.length +
+                scan.getStartRow().length + ROW_KEY_SEPARATOR_BYTE.length + scan.getStopRow().length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, indexHTable.getName().toBytes(), 0, indexHTable.getName().toBytes().length);
+        targetOffset += indexHTable.getName().toBytes().length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, regionName, 0, regionName.length);
+        targetOffset += regionName.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, scan.getStartRow(), 0, scan.getStartRow().length);
+        targetOffset += scan.getStartRow().length;
 
 Review comment:
   Nothing. The separator is just for helping the human debugger copy subfields within a row key. @priyankporwal suggested this separator. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] priyankporwal commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
priyankporwal commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395850243
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + regionName.length + ROW_KEY_SEPARATOR_BYTE.length +
+                scan.getStartRow().length + ROW_KEY_SEPARATOR_BYTE.length + scan.getStopRow().length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, indexHTable.getName().toBytes(), 0, indexHTable.getName().toBytes().length);
+        targetOffset += indexHTable.getName().toBytes().length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, regionName, 0, regionName.length);
+        targetOffset += regionName.length;
+        Bytes.putBytes(rowKey, targetOffset, ROW_KEY_SEPARATOR_BYTE, 0, ROW_KEY_SEPARATOR_BYTE.length);
+        targetOffset += ROW_KEY_SEPARATOR_BYTE.length;
+        Bytes.putBytes(rowKey, targetOffset, scan.getStartRow(), 0, scan.getStartRow().length);
+        targetOffset += scan.getStartRow().length;
 
 Review comment:
   Valid points valid on both sides. I suggested a separator and pipe character as an aid for humans reading these keys. Although not perfect, it will work for programmatic access as well, assuming timestamp and index-names cannot have '|' character - by looking at first 2 occurrences only. The key may have '|' but that's a problem for another day.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395388535
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
 
 Review comment:
    I will extract the code out to a separate method. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395318526
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
+        // The row key for the result table : timestamp | index table name | datable table region name |
+        //                                    scan start row | scan stop row
+        byte[] rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
 
 Review comment:
   Would be good to extract row key building into its own function

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde merged pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde merged pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395323398
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -257,16 +258,29 @@ private void logToIndexToolResultTable() throws IOException {
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] regionName = Bytes.toBytes(region.getRegionInfo().getRegionNameAsString());
-        // The row key for the result table is the max timestamp of the scan + the table region name + scan start row
-        // + scan stop row
-        byte[] rowKey = new byte[keyPrefix.length + regionName.length + scan.getStartRow().length +
-                scan.getStopRow().length];
-        Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        Bytes.putBytes(rowKey, keyPrefix.length, regionName, 0, regionName.length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length, scan.getStartRow(), 0,
-                scan.getStartRow().length);
-        Bytes.putBytes(rowKey, keyPrefix.length + regionName.length + scan.getStartRow().length,
-                scan.getStopRow(), 0, scan.getStopRow().length);
+        int targetOffset = 0;
 
 Review comment:
   If you don't have the bandwidth for the read part, I'm happy to help out with it, either in this JIRA or another. If it doesn't get put into Phoenix here it's code I have to write anyway. :-)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
kadirozde commented on issue #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#issuecomment-602704176
 
 
   > @kadirozde - I offered (and am still offering) to split out the code into a separate class and write read / field parsing logic in a separate JIRA, but I believe @swaroopak 's request for a test to cover the write logic you've changed is still needed for this JIRA.
   
   @gjacoby126, the change is covered by many existing tests already. You cannot get verification counters correct if the key is not constructed properly. As I said before, this JIRA does not offer anything for parsing or extracting subfields within a row key. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [phoenix] gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…

Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #737: PHOENIX-5773 Index tool output tables should support multiple simulta…
URL: https://github.com/apache/phoenix/pull/737#discussion_r395319574
 
 

 ##########
 File path: phoenix-core/src/main/java/org/apache/phoenix/coprocessor/IndexRebuildRegionScanner.java
 ##########
 @@ -412,30 +426,31 @@ public void logToIndexToolOutputTable(byte[] dataRowKey, byte[] indexRowKey, lon
         long scanMaxTs = scan.getTimeRange().getMax();
         byte[] keyPrefix = Bytes.toBytes(Long.toString(scanMaxTs));
         byte[] rowKey;
-        // The row key for the output table is the max timestamp of the scan + data row key
-        if (dataRowKey != null) {
-            rowKey = new byte[keyPrefix.length + dataRowKey.length];
-            Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-            Bytes.putBytes(rowKey, keyPrefix.length, dataRowKey, 0, dataRowKey.length);
-        } else {
-            rowKey = new byte[keyPrefix.length];
-            Bytes.putBytes(rowKey, 0, keyPrefix, 0, keyPrefix.length);
-        }
+        int targetOffset = 0;
+        // The row key for the output table : timestamp | index table name | data row key
+        rowKey = new byte[keyPrefix.length + ROW_KEY_SEPARATOR_BYTE.length + indexHTable.getName().toBytes().length +
+                ROW_KEY_SEPARATOR_BYTE.length + dataRowKey.length];
+        Bytes.putBytes(rowKey, targetOffset, keyPrefix, 0, keyPrefix.length);
+        targetOffset += keyPrefix.length;
 
 Review comment:
   likewise, extracting to a method to build the output table row key would be good. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services