You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:35:06 UTC
[GitHub] [phoenix] sandeepvinayak opened a new pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
sandeepvinayak opened a new pull request #951:
URL: https://github.com/apache/phoenix/pull/951
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] sandeepvinayak edited a comment on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
sandeepvinayak edited a comment on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-720270639
@gjacoby126 can you please review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-725728633
@sandeepvinayak - could you please rename the commit to include the JIRA number? (Might as well squash while you're rebasing too.) Then I'll commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-720373974
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 25s | master passed |
| +1 :green_heart: | compile | 1m 1s | master passed |
| +1 :green_heart: | checkstyle | 0m 32s | master passed |
| +1 :green_heart: | javadoc | 0m 47s | master passed |
| +0 :ok: | spotbugs | 3m 9s | phoenix-core in master has 969 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 14s | the patch passed |
| +1 :green_heart: | compile | 0m 58s | the patch passed |
| +1 :green_heart: | javac | 0m 58s | the patch passed |
| -1 :x: | checkstyle | 0m 32s | phoenix-core: The patch generated 8 new + 13 unchanged - 3 fixed = 21 total (was 16) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 20s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 171m 31s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 210m 27s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.TableSnapshotReadsMapReduceIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/951 |
| JIRA Issue | PHOENIX-5895 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux ba3e57d1202d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / e828ef7 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/1/testReport/ |
| Max. process+thread count | 6044 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/1/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-721690093
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 39s | master passed |
| +1 :green_heart: | compile | 0m 59s | master passed |
| +1 :green_heart: | checkstyle | 0m 30s | master passed |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
| +0 :ok: | spotbugs | 2m 54s | phoenix-core in master has 967 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 7m 25s | the patch passed |
| +1 :green_heart: | compile | 0m 53s | the patch passed |
| +1 :green_heart: | javac | 0m 53s | the patch passed |
| -1 :x: | checkstyle | 0m 28s | phoenix-core: The patch generated 1 new + 10 unchanged - 6 fixed = 11 total (was 16) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 43s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 7s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 97m 7s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
| | | 130m 57s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | phoenix.end2end.OrphanViewToolIT |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/3/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/951 |
| JIRA Issue | PHOENIX-5895 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 062233f83100 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 337d795 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/3/testReport/ |
| Max. process+thread count | 6509 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/3/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-721626671
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 38s | master passed |
| +1 :green_heart: | compile | 0m 57s | master passed |
| +1 :green_heart: | checkstyle | 0m 29s | master passed |
| +1 :green_heart: | javadoc | 0m 44s | master passed |
| +0 :ok: | spotbugs | 2m 55s | phoenix-core in master has 967 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 7m 20s | the patch passed |
| +1 :green_heart: | compile | 0m 52s | the patch passed |
| +1 :green_heart: | javac | 0m 52s | the patch passed |
| -1 :x: | checkstyle | 0m 31s | phoenix-core: The patch generated 2 new + 10 unchanged - 6 fixed = 12 total (was 16) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 43s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 4s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 95m 10s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| | | 128m 55s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/951 |
| JIRA Issue | PHOENIX-5895 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux 6dd01d94eb77 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 337d795 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/2/testReport/ |
| Max. process+thread count | 6726 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/2/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-725736215
Thanks, @sandeepvinayak
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 merged pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
gjacoby126 merged pull request #951:
URL: https://github.com/apache/phoenix/pull/951
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] sandeepvinayak commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-720270639
@gjacoby126
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] stoty commented on pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
stoty commented on pull request #951:
URL: https://github.com/apache/phoenix/pull/951#issuecomment-725788578
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 12m 45s | master passed |
| +1 :green_heart: | compile | 0m 58s | master passed |
| +1 :green_heart: | checkstyle | 0m 30s | master passed |
| +1 :green_heart: | javadoc | 0m 43s | master passed |
| +0 :ok: | spotbugs | 2m 58s | phoenix-core in master has 962 extant spotbugs warnings. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 9m 35s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| -1 :x: | checkstyle | 0m 35s | phoenix-core: The patch generated 1 new + 10 unchanged - 6 fixed = 11 total (was 16) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 51s | the patch passed |
| +1 :green_heart: | spotbugs | 3m 44s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 104m 38s | phoenix-core in the patch failed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 141m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/4/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/phoenix/pull/951 |
| JIRA Issue | PHOENIX-5895 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile |
| uname | Linux fb17ef8c9f45 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev/phoenix-personality.sh |
| git revision | master / 8bb8956 |
| Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 |
| checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt |
| unit | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/4/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt |
| Test Results | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/4/testReport/ |
| Max. process+thread count | 6855 (vs. ulimit of 30000) |
| modules | C: phoenix-core U: phoenix-core |
| Console output | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-951/4/console |
| versions | git=2.7.4 maven=3.3.9 spotbugs=4.1.3 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] sandeepvinayak commented on a change in pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
sandeepvinayak commented on a change in pull request #951:
URL: https://github.com/apache/phoenix/pull/951#discussion_r517140962
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
##########
@@ -35,30 +36,30 @@
* be copied. This WALEntryFilter will only allow tenant-owned rows in SYSTEM.CATALOG to
* be replicated. Data from all other tables is automatically passed.
*/
-public class SystemCatalogWALEntryFilter implements WALEntryFilter {
+public class SystemCatalogWALEntryFilter implements WALEntryFilter, WALCellFilter {
+ // This is an optimization to just skip the cell filter if we do not care about
+ // cell filter for certain WALEdits.
+ private boolean skipCellFilter;
@Override
public WAL.Entry filter(WAL.Entry entry) {
-
+ // We use the WALCellFilter to filter the cells from entry, WALEntryFilter
+ // should not block anything
//if the WAL.Entry's table isn't System.Catalog or System.Child_Link, it auto-passes this filter
- //TODO: when Phoenix drops support for pre-1.3 versions of HBase, redo as a WALCellFilter
if (!SchemaUtil.isMetaTable(entry.getKey().getTableName().getName())){
- return entry;
+ skipCellFilter = true;
+ } else {
+ skipCellFilter = false;
}
+ return entry;
+ }
- List<Cell> cells = entry.getEdit().getCells();
- List<Cell> cellsToRemove = Lists.newArrayList();
- for (Cell cell : cells) {
- if (!isTenantRowCell(cell)){
- cellsToRemove.add(cell);
- }
- }
- cells.removeAll(cellsToRemove);
- if (cells.size() > 0) {
- return entry;
- } else {
- return null;
+ @Override
+ public Cell filterCell(WAL.Entry entry, Cell cell) {
+ if (skipCellFilter) {
+ return cell;
}
+ return isTenantRowCell(cell) ? cell : null;
}
private boolean isTenantRowCell(Cell cell) {
Review comment:
yes, it should be same, I will change that
```java
public byte [] get() {
if (this.bytes == null) {
throw new IllegalStateException("Uninitialiized. Null constructor " +
"called w/o accompaying readFields invocation");
}
return this.bytes;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] gjacoby126 commented on a change in pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
gjacoby126 commented on a change in pull request #951:
URL: https://github.com/apache/phoenix/pull/951#discussion_r516334881
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilterIT.java
##########
@@ -134,11 +135,16 @@ public void testSystemCatalogWALEntryFilter() throws Exception {
//verify that the tenant view WAL.Entry passes the filter and the non-tenant view does not
SystemCatalogWALEntryFilter filter = new SystemCatalogWALEntryFilter();
- Assert.assertNull(filter.filter(nonTenantEntry));
- WAL.Entry filteredTenantEntry = filter.filter(tenantEntry);
+ // Chain the system catalog WAL entry filter to ChainWALEntryFilter
+ ChainWALEntryFilter chainWALEntryFilter = new ChainWALEntryFilter(filter);
+ // Asserting the WALEdit for non tenant has cells before getting filtered
+ Assert.assertTrue(nonTenantEntry.getEdit().size() > 0);
+ // All the cells will get removed by the filter since they do not belong to tenant
+ Assert.assertTrue("Non tenant edits for system catalog got ", chainWALEntryFilter.filter(nonTenantEntry).getEdit().isEmpty());
Review comment:
nit: system catalog get
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
##########
@@ -35,30 +36,30 @@
* be copied. This WALEntryFilter will only allow tenant-owned rows in SYSTEM.CATALOG to
* be replicated. Data from all other tables is automatically passed.
*/
-public class SystemCatalogWALEntryFilter implements WALEntryFilter {
+public class SystemCatalogWALEntryFilter implements WALEntryFilter, WALCellFilter {
+ // This is an optimization to just skip the cell filter if we do not care about
+ // cell filter for certain WALEdits.
+ private boolean skipCellFilter;
@Override
public WAL.Entry filter(WAL.Entry entry) {
-
+ // We use the WALCellFilter to filter the cells from entry, WALEntryFilter
+ // should not block anything
//if the WAL.Entry's table isn't System.Catalog or System.Child_Link, it auto-passes this filter
- //TODO: when Phoenix drops support for pre-1.3 versions of HBase, redo as a WALCellFilter
if (!SchemaUtil.isMetaTable(entry.getKey().getTableName().getName())){
- return entry;
+ skipCellFilter = true;
+ } else {
+ skipCellFilter = false;
}
+ return entry;
+ }
- List<Cell> cells = entry.getEdit().getCells();
- List<Cell> cellsToRemove = Lists.newArrayList();
- for (Cell cell : cells) {
- if (!isTenantRowCell(cell)){
- cellsToRemove.add(cell);
- }
- }
- cells.removeAll(cellsToRemove);
- if (cells.size() > 0) {
- return entry;
- } else {
- return null;
+ @Override
+ public Cell filterCell(WAL.Entry entry, Cell cell) {
+ if (skipCellFilter) {
+ return cell;
}
+ return isTenantRowCell(cell) ? cell : null;
}
private boolean isTenantRowCell(Cell cell) {
Review comment:
@virajjasani - yes, I believe that will work. ImmutableBytesWritable.get() just returns the underlying byte array
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [phoenix] virajjasani commented on a change in pull request #951: PHOENIX-5895: Leverage WALCellFilter in the SystemCatalogWALEntryFilter to replicate system catalog table
Posted by GitBox <gi...@apache.org>.
virajjasani commented on a change in pull request #951:
URL: https://github.com/apache/phoenix/pull/951#discussion_r515785773
##########
File path: phoenix-core/src/it/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilterIT.java
##########
@@ -134,11 +135,16 @@ public void testSystemCatalogWALEntryFilter() throws Exception {
//verify that the tenant view WAL.Entry passes the filter and the non-tenant view does not
SystemCatalogWALEntryFilter filter = new SystemCatalogWALEntryFilter();
- Assert.assertNull(filter.filter(nonTenantEntry));
- WAL.Entry filteredTenantEntry = filter.filter(tenantEntry);
+ // Chain the system catalog WAL entry filter to ChainWALEntryFilter
+ ChainWALEntryFilter chainWALEntryFilter = new ChainWALEntryFilter(filter);
+ // Asserting the WALEdit for non tenant has cells before getting filtered
+ Assert.assertTrue(nonTenantEntry.getEdit().size() > 0);
+ // All the cells will get removed by the filter since they do not belong to tenant
+ Assert.assertTrue("Non tenant edits for system catalog got ", chainWALEntryFilter.filter(nonTenantEntry).getEdit().isEmpty());
+ WAL.Entry filteredTenantEntry = chainWALEntryFilter.filter(tenantEntry);
Assert.assertNotNull("Tenant view was filtered when it shouldn't be!", filteredTenantEntry);
Assert.assertEquals(tenantEntry.getEdit().size(),
- filter.filter(tenantEntry).getEdit().size());
+ chainWALEntryFilter.filter(tenantEntry).getEdit().size());
Review comment:
nit: this could be `filteredTenantEntry.getEdit().size())` ?
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
##########
@@ -35,30 +36,30 @@
* be copied. This WALEntryFilter will only allow tenant-owned rows in SYSTEM.CATALOG to
* be replicated. Data from all other tables is automatically passed.
*/
-public class SystemCatalogWALEntryFilter implements WALEntryFilter {
+public class SystemCatalogWALEntryFilter implements WALEntryFilter, WALCellFilter {
+ // This is an optimization to just skip the cell filter if we do not care about
+ // cell filter for certain WALEdits.
+ private boolean skipCellFilter;
@Override
public WAL.Entry filter(WAL.Entry entry) {
-
+ // We use the WALCellFilter to filter the cells from entry, WALEntryFilter
+ // should not block anything
//if the WAL.Entry's table isn't System.Catalog or System.Child_Link, it auto-passes this filter
- //TODO: when Phoenix drops support for pre-1.3 versions of HBase, redo as a WALCellFilter
if (!SchemaUtil.isMetaTable(entry.getKey().getTableName().getName())){
- return entry;
+ skipCellFilter = true;
+ } else {
+ skipCellFilter = false;
}
+ return entry;
+ }
- List<Cell> cells = entry.getEdit().getCells();
- List<Cell> cellsToRemove = Lists.newArrayList();
- for (Cell cell : cells) {
- if (!isTenantRowCell(cell)){
- cellsToRemove.add(cell);
- }
- }
- cells.removeAll(cellsToRemove);
- if (cells.size() > 0) {
- return entry;
- } else {
- return null;
+ @Override
+ public Cell filterCell(WAL.Entry entry, Cell cell) {
+ if (skipCellFilter) {
+ return cell;
}
+ return isTenantRowCell(cell) ? cell : null;
}
private boolean isTenantRowCell(Cell cell) {
Review comment:
Not related to this changes, but wondering if we need to construct ImmutableBytesWritable object.
Can this method impl be reduced to:
```
return cell.getRowArray()[cell.getRowOffset()] != QueryConstants.SEPARATOR_BYTE;
```
##########
File path: phoenix-core/src/main/java/org/apache/phoenix/replication/SystemCatalogWALEntryFilter.java
##########
@@ -21,6 +21,7 @@
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.replication.WALCellFilter;
import org.apache.hadoop.hbase.replication.WALEntryFilter;
import org.apache.hadoop.hbase.wal.WAL;
import org.apache.phoenix.query.QueryConstants;
Review comment:
nit: we can get rid of `List` and `Lists` imports
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org