You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2021/02/18 10:40:40 UTC
[GitHub] [phoenix] stoty opened a new pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
stoty opened a new pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#issuecomment-783197142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 15m 31s | master passed |
| +0 | hbaserecompile | 22m 5s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 7s | master passed |
| +1 :green_heart: | checkstyle | 0m 36s | master passed |
| +1 :green_heart: | javadoc | 0m 53s | master passed |
| +0 :ok: | spotbugs | 3m 6s | phoenix-core in master has 959 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 14s | the patch passed |
| +0 | hbaserecompile | 18m 13s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| -1 :x: | checkstyle | 0m 37s | phoenix-core: The patch generated 24 new + 138 unchanged - 4 fixed = 162 total (was 142) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 20s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 107m 14s | phoenix-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 158m 44s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/1144 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile |
| uname | Linux ea68963707d2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 34113c5 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/4/artifact/yetus-general-check/output/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/4/testReport/ |
| Max. process+thread count | 10892 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/4/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty closed pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
stoty closed pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] virajjasani commented on a change in pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#discussion_r579814684
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
Review comment:
This should solve the problem of verifying index rows right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on a change in pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
stoty commented on a change in pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#discussion_r579992339
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/CsvBulkLoadToolIT.java
##########
@@ -36,16 +39,30 @@
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Result;
Review comment:
OK
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
Review comment:
OK
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
+
+ public IndexStatusUpdater(byte[] emptyKeyValueCF, byte[] emptyKeyValueQualifier) {
+ this.emptyKeyValueCF = emptyKeyValueCF;
+ this.emptyKeyValueQualifier = emptyKeyValueQualifier;
+ emptyKeyValueCFLength = emptyKeyValueCF.length;
+ emptyKeyValueQualifierLength = emptyKeyValueQualifier.length;
+ }
+
+ /**
+ * Update the Empty cell values to VERIFIED in the passed keyValues list
+ *
+ * @param keyValues will be modified
+ */
+ public void setVerfied(List<Cell> keyValues) {
+ for(int i=0; i < keyValues.size() ; i++) {
+ Cell kv = keyValues.get(i);
Review comment:
I purposefully chose this construct, as we're processing a lot a objects, and this is reportedly faster and easier on memory / GC.
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
for (Cell kv : keyValueList) {
- list.add(kv);
+ cellsForTable.add(kv);
}
Review comment:
OK
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/CsvBulkLoadToolIT.java
##########
@@ -318,6 +335,27 @@ public void testImportWithIndex() throws Exception {
rs.close();
stmt.close();
+
+ checkIndexTableIsVerfied("TABLE3_IDX");
+ }
+
+ private void checkIndexTableIsVerfied(String indexTableName) throws SQLException, IOException {
Review comment:
Thanks
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
Review comment:
Strictly speaking, we are not verifying them, as we're generating the records in bulk, and we assume that they are correct (we assume that the bluk load job runs fully).
We're just setting the status to verified to avoid repairing the freshly generated index rows.
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
+
+ public IndexStatusUpdater(byte[] emptyKeyValueCF, byte[] emptyKeyValueQualifier) {
+ this.emptyKeyValueCF = emptyKeyValueCF;
+ this.emptyKeyValueQualifier = emptyKeyValueQualifier;
+ emptyKeyValueCFLength = emptyKeyValueCF.length;
+ emptyKeyValueQualifierLength = emptyKeyValueQualifier.length;
+ }
+
+ /**
+ * Update the Empty cell values to VERIFIED in the passed keyValues list
+ *
+ * @param keyValues will be modified
+ */
+ public void setVerfied(List<Cell> keyValues) {
Review comment:
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] virajjasani commented on a change in pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#discussion_r579813022
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
+
+ public IndexStatusUpdater(byte[] emptyKeyValueCF, byte[] emptyKeyValueQualifier) {
+ this.emptyKeyValueCF = emptyKeyValueCF;
+ this.emptyKeyValueQualifier = emptyKeyValueQualifier;
+ emptyKeyValueCFLength = emptyKeyValueCF.length;
+ emptyKeyValueQualifierLength = emptyKeyValueQualifier.length;
+ }
+
+ /**
+ * Update the Empty cell values to VERIFIED in the passed keyValues list
+ *
+ * @param keyValues will be modified
+ */
+ public void setVerfied(List<Cell> keyValues) {
Review comment:
nit: `"s/setVerfied/setVerified"`
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
Review comment:
nit: good to mark all `final`
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/CsvBulkLoadToolIT.java
##########
@@ -318,6 +335,27 @@ public void testImportWithIndex() throws Exception {
rs.close();
stmt.close();
+
+ checkIndexTableIsVerfied("TABLE3_IDX");
+ }
+
+ private void checkIndexTableIsVerfied(String indexTableName) throws SQLException, IOException {
Review comment:
nit: `"s/checkIndexTableIsVerfied/checkIndexTableIsVerified"`
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -411,8 +421,46 @@ public void errorOnRecord(T record, Throwable throwable) {
ImportPreUpsertKeyValueProcessor {
@Override
- public List<Cell> preUpsert(byte[] rowKey, List<Cell> keyValues) {
+ public List<Cell> preUpsert(byte[] tableName, List<Cell> keyValues) {
return keyValues;
}
}
+
+ /**
+ * Updates the Empty cell value to VERIFIED for global index table rows
+ */
+ private static class IndexStatusUpdater {
+
+ private byte[] emptyKeyValueCF;
+ private int emptyKeyValueCFLength;
+ private byte[] emptyKeyValueQualifier;
+ private int emptyKeyValueQualifierLength;
+
+ public IndexStatusUpdater(byte[] emptyKeyValueCF, byte[] emptyKeyValueQualifier) {
+ this.emptyKeyValueCF = emptyKeyValueCF;
+ this.emptyKeyValueQualifier = emptyKeyValueQualifier;
+ emptyKeyValueCFLength = emptyKeyValueCF.length;
+ emptyKeyValueQualifierLength = emptyKeyValueQualifier.length;
+ }
+
+ /**
+ * Update the Empty cell values to VERIFIED in the passed keyValues list
+ *
+ * @param keyValues will be modified
+ */
+ public void setVerfied(List<Cell> keyValues) {
+ for(int i=0; i < keyValues.size() ; i++) {
+ Cell kv = keyValues.get(i);
Review comment:
can be replaced by
```
for (Cell kv : keyValues) {
```
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/CsvBulkLoadToolIT.java
##########
@@ -36,16 +39,30 @@
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CompareOperator;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.Result;
Review comment:
Few unused imports, nothing urgent, can be removed while committing changes
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
Review comment:
This should solve the problem right?
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
for (Cell kv : keyValueList) {
- list.add(kv);
+ cellsForTable.add(kv);
}
Review comment:
nit: can be replaced with `cellsForTable.addAll(keyValueList)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] virajjasani commented on a change in pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#discussion_r580015439
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/mapreduce/FormatToBytesWritableMapper.java
##########
@@ -176,17 +179,20 @@ protected void map(LongWritable key, Text value, Context context) throws IOExcep
while (uncommittedDataIterator.hasNext()) {
Pair<byte[], List<Cell>> kvPair = uncommittedDataIterator.next();
List<Cell> keyValueList = kvPair.getSecond();
- keyValueList = preUpdateProcessor.preUpsert(kvPair.getFirst(), keyValueList);
- byte[] first = kvPair.getFirst();
+ byte[] tableName = kvPair.getFirst();
+ keyValueList = preUpdateProcessor.preUpsert(tableName, keyValueList);
// Create a list of KV for each table
for (int i = 0; i < tableNames.size(); i++) {
- if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), first) == 0) {
+ if (Bytes.compareTo(Bytes.toBytes(tableNames.get(i)), tableName) == 0) {
if (!map.containsKey(i)) {
map.put(i, new ArrayList<Cell>());
}
- List<Cell> list = map.get(i);
+ List<Cell> cellsForTable = map.get(i);
+ if(indexStatusUpdaters[i] != null) {
+ indexStatusUpdaters[i].setVerfied(keyValueList);
+ }
Review comment:
Got it, makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#issuecomment-783199632
@virajjasani If the changes look good can you add a formal +1 / approved ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #1144: PHOENIX-6386 Bulkload generates unverified index rows
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #1144:
URL: https://github.com/apache/phoenix/pull/1144#issuecomment-781491427
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 15m 28s | master passed |
| +0 | hbaserecompile | 22m 44s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 6s | master passed |
| +1 :green_heart: | checkstyle | 0m 35s | master passed |
| +1 :green_heart: | javadoc | 0m 52s | master passed |
| +0 :ok: | spotbugs | 3m 6s | phoenix-core in master has 959 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 31s | the patch passed |
| +0 | hbaserecompile | 18m 2s | HBase recompiled. |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| -1 :x: | checkstyle | 0m 36s | phoenix-core: The patch generated 33 new + 138 unchanged - 4 fixed = 171 total (was 142) |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 17s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 237m 17s | phoenix-core in the patch passed. |
| +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. |
| | | 289m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/1144 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile |
| uname | Linux 4500104df372 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 34113c5 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| whitespace | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/3/artifact/yetus-general-check/output/whitespace-eol.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/3/testReport/ |
| Max. process+thread count | 10175 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1144/3/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org