You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/10/21 14:27:48 UTC
[GitHub] [phoenix] richardantal opened a new pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
richardantal opened a new pull request #933:
URL: https://github.com/apache/phoenix/pull/933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r509383290
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/StoreNullsIT.java
##########
@@ -76,6 +81,15 @@
private final String ddlFormat;
private String dataTableName;
+
+ // In this class we depend on the major compaction to be done instantly
+ // so wo are overwriting the PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY to 0
+ @BeforeClass
Review comment:
I would expect the change in TestUtil to fix this case as well.
Why do we need to override PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 closed pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
gjacoby126 closed pull request #933:
URL: https://github.com/apache/phoenix/pull/933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r509404314
##########
File path: phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java
##########
@@ -840,10 +840,16 @@ public static void doMajorCompaction(Connection conn, String tableName) throws E
byte[] markerRowKey = Bytes.toBytes("TO_DELETE");
Put put = new Put(markerRowKey);
- put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
+ long timestamp = 0;
+ // We do not want to wait an hour because of PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY
+ // So set the timestamp of the put and delete as early as possible
+ put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.EMPTY_COLUMN_VALUE_BYTES, timestamp,
+ QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
htable.put(put);
Delete delete = new Delete(markerRowKey);
- delete.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
+ delete.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
Review comment:
Maybe use timestamp + 1 here, to make sure that delete comes after the insert ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] richardantal commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
richardantal commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r510012836
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/StoreNullsIT.java
##########
@@ -76,6 +81,15 @@
private final String ddlFormat;
private String dataTableName;
+
+ // In this class we depend on the major compaction to be done instantly
+ // so wo are overwriting the PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY to 0
+ @BeforeClass
Review comment:
But the comment here is not accurate, thats right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty edited a comment on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty edited a comment on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-716102115
Opened a PR for the addendum at https://github.com/apache/phoenix/pull/939 so that the precommit tests get executed on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-715401296
StoreNullsIT fails when running as part of the full IT suite.
Probably needs to be switched to @NeedsOwnCluster
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty closed pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty closed pull request #933:
URL: https://github.com/apache/phoenix/pull/933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r509877509
##########
File path: phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java
##########
@@ -840,10 +840,16 @@ public static void doMajorCompaction(Connection conn, String tableName) throws E
byte[] markerRowKey = Bytes.toBytes("TO_DELETE");
Put put = new Put(markerRowKey);
- put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
+ long timestamp = 0;
+ // We do not want to wait an hour because of PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY
+ // So set the timestamp of the put and delete as early as possible
+ put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
+ QueryConstants.EMPTY_COLUMN_VALUE_BYTES, timestamp,
+ QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
htable.put(put);
Delete delete = new Delete(markerRowKey);
- delete.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
+ delete.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
Review comment:
Not needed, identical timestamp will be deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-716102115
Opened a PR for the addendum at https://github.com/apache/phoenix/pull/938 so that the precommit tests get executed on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-715127627
Addressed Geoffrey's NIT, and committed to fix the build ASAP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-713758320
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 6m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ 4.x Compile Tests _ |
| +1 :green_heart: | mvninstall | 11m 44s | 4.x passed |
| +1 :green_heart: | compile | 0m 59s | 4.x passed |
| +1 :green_heart: | checkstyle | 0m 30s | 4.x passed |
| +1 :green_heart: | javadoc | 0m 46s | 4.x passed |
| +0 :ok: | spotbugs | 3m 8s | phoenix-core in 4.x has 956 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 6s | the patch passed |
| +1 :green_heart: | compile | 1m 0s | the patch passed |
| +1 :green_heart: | javac | 1m 0s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 46s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 20s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 174m 49s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 212m 53s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.TenantSpecificViewIndexSaltedIT |
| | phoenix.end2end.index.ImmutableIndexExtendedIT |
| | phoenix.end2end.StoreNullsIT |
| | phoenix.end2end.index.GlobalImmutableTxIndexIT |
| | phoenix.end2end.join.HashJoinPersistentCacheIT |
| | phoenix.end2end.IndexRepairRegionScannerIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/933 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 0bc745df6e7a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | 4.x / 2ff920a |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/1/testReport/ |
| Max. process+thread count | 5930 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r510479056
##########
File path: phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java
##########
@@ -840,10 +840,16 @@ public static void doMajorCompaction(Connection conn, String tableName) throws E
byte[] markerRowKey = Bytes.toBytes("TO_DELETE");
Put put = new Put(markerRowKey);
- put.addColumn(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES, QueryConstants.EMPTY_COLUMN_VALUE_BYTES);
+ long timestamp = 0;
Review comment:
nit: 0L
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] richardantal commented on a change in pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
richardantal commented on a change in pull request #933:
URL: https://github.com/apache/phoenix/pull/933#discussion_r510010055
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/end2end/StoreNullsIT.java
##########
@@ -76,6 +81,15 @@
private final String ddlFormat;
private String dataTableName;
+
+ // In this class we depend on the major compaction to be done instantly
+ // so wo are overwriting the PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY to 0
+ @BeforeClass
Review comment:
My change in TestUtil solves the hanging tests, because there we waited until a put and delete is deleted, which never happened because they were just done before calling the major compaction with a PHOENIX_MAX_LOOKBACK_AGE of an hour.
In StoreNullsIT in testDeletes, we upsert a line into a table, delete it run the major compaction.
Then open a historical connection after the upsert but before the delete, previously when PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY was set to 0 we could not read back the data, but if it is set to an hour, the major compaction will not delete that. So the test would fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-714515577
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 52s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ 4.x Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 35s | 4.x passed |
| +1 :green_heart: | compile | 1m 6s | 4.x passed |
| +1 :green_heart: | checkstyle | 0m 33s | 4.x passed |
| +1 :green_heart: | javadoc | 0m 55s | 4.x passed |
| +0 :ok: | spotbugs | 3m 57s | phoenix-core in 4.x has 954 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 7m 33s | the patch passed |
| +1 :green_heart: | compile | 1m 13s | the patch passed |
| +1 :green_heart: | javac | 1m 13s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed |
| +1 :green_heart: | spotbugs | 4m 8s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 192m 3s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 230m 36s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.UngroupedIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/933 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 6cd439b8389a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | 4.x / afda3c1 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/2/testReport/ |
| Max. process+thread count | 5582 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-933/2/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on pull request #933: PHOENIX-6197 AggregateIT and StoreNullsIT hangs
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on pull request #933:
URL: https://github.com/apache/phoenix/pull/933#issuecomment-730851022
JIRA is resolved, closing PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org