You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/10/14 18:28:46 UTC
[GitHub] [phoenix] m2je opened a new pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
m2je opened a new pull request #921:
URL: https://github.com/apache/phoenix/pull/921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] m2je commented on pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
Posted by GitBox <gi...@apache.org>.
m2je commented on pull request #921:
URL: https://github.com/apache/phoenix/pull/921#issuecomment-731635976
> See above comments
@gjacoby126 Sorry for the late turn around. Please take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] m2je commented on a change in pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
Posted by GitBox <gi...@apache.org>.
m2je commented on a change in pull request #921:
URL: https://github.com/apache/phoenix/pull/921#discussion_r528242126
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/hbase/index/Indexer.java
##########
@@ -311,7 +311,17 @@ public Result preIncrementAfterRowLock(final ObserverContext<RegionCoprocessorEn
if (!mutations.isEmpty()) {
Region region = e.getEnvironment().getRegion();
// Otherwise, submit the mutations directly here
- region.batchMutate(mutations.toArray(new Mutation[0]));
+ OperationStatus[] batchMutationStatus = region.batchMutate(
Review comment:
Done
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
##########
@@ -118,6 +122,33 @@ else if (cause == null || cause instanceof IOException) {
}
}
+ /**
+ * Converts a batch operation status into an IOException if the operation failed partially or completely. In cases
+ * of multiple failures, non-retriable errors are prioritized. If any part of the batch operation failed in a way
+ * that isn't retriable, an instance of the corresponding DoNotRetryIOException subtype is returned, even if other
+ * retriable errors are present. If all parts were completed successfully, null will be returned.
+ */
+ public static IOException createIOException(OperationStatus[] batchOperationStatus) {
+ IOException retriableException = null;
+ for (int i = 0; i < batchOperationStatus.length; i++) {
+ OperationStatus status = batchOperationStatus[i];
+ switch (status.getOperationStatusCode()) {
+ case SUCCESS:
+ break;
+ case BAD_FAMILY:
+ return new NoSuchColumnFamilyException(status.getExceptionMsg());
+ case SANITY_CHECK_FAILURE:
+ return new FailedSanityCheckException(status.getExceptionMsg());
+ case STORE_TOO_BUSY:
+ retriableException = new RegionTooBusyException(status.getExceptionMsg());
+ break;
+ default:
+ return new DoNotRetryIOException(status.getExceptionMsg());
Review comment:
By looking at Hbase's code it does seem that `FAILURE` is not a retriable state, [example](https://github.com/apache/hbase/blob/branch-2.1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3765)
Added a retry for `NOT_RUN` (which if I understand correctly is used when mutation was never executed)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #921:
URL: https://github.com/apache/phoenix/pull/921#issuecomment-731657197
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 38s | master passed |
| +1 :green_heart: | compile | 1m 0s | master passed |
| +1 :green_heart: | checkstyle | 0m 45s | master passed |
| +1 :green_heart: | javadoc | 0m 48s | master passed |
| +0 :ok: | spotbugs | 3m 8s | phoenix-core in master has 966 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 12s | the patch passed |
| +1 :green_heart: | compile | 0m 55s | the patch passed |
| +1 :green_heart: | javac | 0m 55s | the patch passed |
| -1 :x: | checkstyle | 0m 46s | phoenix-core: The patch generated 23 new + 887 unchanged - 5 fixed = 910 total (was 892) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 22s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 162m 7s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
| | | 201m 32s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.join.SubqueryUsingSortMergeJoinIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/921 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 719fb8347180 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / f1a0860 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/1/testReport/ |
| Max. process+thread count | 6156 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #921:
URL: https://github.com/apache/phoenix/pull/921#issuecomment-731693717
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 40s | master passed |
| +1 :green_heart: | compile | 1m 2s | master passed |
| +1 :green_heart: | checkstyle | 0m 45s | master passed |
| +1 :green_heart: | javadoc | 0m 48s | master passed |
| +0 :ok: | spotbugs | 3m 9s | phoenix-core in master has 966 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 58s | the patch passed |
| +1 :green_heart: | javac | 0m 58s | the patch passed |
| -1 :x: | checkstyle | 0m 46s | phoenix-core: The patch generated 22 new + 887 unchanged - 5 fixed = 909 total (was 892) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 20s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 159m 24s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| | | 199m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/921 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 98e822388ab6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / f1a0860 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/2/testReport/ |
| Max. process+thread count | 6114 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-921/2/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #921: PHOENIX-6187 Avoid swallowing UPSERT... ON DUPLICATION KEY UPDATE failures
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #921:
URL: https://github.com/apache/phoenix/pull/921#discussion_r510488491
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/hbase/index/Indexer.java
##########
@@ -311,7 +311,17 @@ public Result preIncrementAfterRowLock(final ObserverContext<RegionCoprocessorEn
if (!mutations.isEmpty()) {
Region region = e.getEnvironment().getRegion();
// Otherwise, submit the mutations directly here
- region.batchMutate(mutations.toArray(new Mutation[0]));
+ OperationStatus[] batchMutationStatus = region.batchMutate(
Review comment:
Please also add to IndexRegionObserver, which is the coproc the new global index framework uses.
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
##########
@@ -118,6 +122,33 @@ else if (cause == null || cause instanceof IOException) {
}
}
+ /**
+ * Converts a batch operation status into an IOException if the operation failed partially or completely. In cases
+ * of multiple failures, non-retriable errors are prioritized. If any part of the batch operation failed in a way
+ * that isn't retriable, an instance of the corresponding DoNotRetryIOException subtype is returned, even if other
+ * retriable errors are present. If all parts were completed successfully, null will be returned.
+ */
+ public static IOException createIOException(OperationStatus[] batchOperationStatus) {
+ IOException retriableException = null;
+ for (int i = 0; i < batchOperationStatus.length; i++) {
+ OperationStatus status = batchOperationStatus[i];
+ switch (status.getOperationStatusCode()) {
+ case SUCCESS:
+ break;
+ case BAD_FAMILY:
+ return new NoSuchColumnFamilyException(status.getExceptionMsg());
+ case SANITY_CHECK_FAILURE:
+ return new FailedSanityCheckException(status.getExceptionMsg());
+ case STORE_TOO_BUSY:
+ retriableException = new RegionTooBusyException(status.getExceptionMsg());
+ break;
+ default:
+ return new DoNotRetryIOException(status.getExceptionMsg());
Review comment:
What about OperationStatus case FAILURE? There are other retriable exceptions other than RegionTooBusyException, aren't there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org