You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sourabh Goyal (Code Review)" <ge...@cloudera.org> on 2021/06/10 17:34:40 UTC
[Impala-ASF-CR] Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Sourabh Goyal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17576
Change subject: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests to address this change in functionality
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 399 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/1
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 23:
Thank you @Vihang and @Kishen for the review.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 18:20:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 20:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9274/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 20
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 20:03:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7353/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 12
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 18:50:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2348
PS19, Line 2348: eventType.equals(DropDatabaseEvent.DROP_DATABASE_EVENT_TYPE) ||
> Alter_Table event is generated for truncate_table. As of now, we just inval
jfyi truncate table does not delete the table or its partitions. It only deletes the files within the partitions. Since there are no partitions deleted we don't need to add anything to deleteEventLog.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 21:08:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#4).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 420 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/4
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#9).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Testing:
1. Added new unit test
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 171 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/9
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
(3 comments)
The patch looks good to me. Thanks for making all the suggested changes. I just have some code styling nits. I can merge the patch once they are addressed.
http://gerrit.cloudera.org:8080/#/c/17576/17/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/17/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2967
PS17, Line 2967: )
: throws MetaExceptio
nit, can go in the same line above. I would suggest you to have your IDE set up with the code-style format from https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml which works well for me.
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@861
PS18, Line 861: this
pls update the java doc
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@865
PS18, Line 865: if (catalogDb
this can go into the above line now.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Aug 2021 23:59:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#16).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 327 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/16
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9170/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 12:34:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2519
PS10, Line 2519: incompleteTable.setCreateEventId(existingTbl.getCreateEventId());
@Vihang: This was probably missed in HANDLE create/drop patch. Please take a look.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 11:30:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9198/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jul 2021 06:42:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7351/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 12:04:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22:
This build failed because of the following test
metadata/test_hms_integration.py::TestHmsIntegrationSanity::test_sanity[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] FAILED
The test is unrelated to this patch.
> Patch Set 22: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7384/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 17:23:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/8911/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jun 2021 23:58:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 17:24:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#13).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Consider a scenario where a table/db already exists and a user drops it via
catalog metastore server endpoint. When recreating the same via Impala
Shell, user gets an error that table/db already exists. This patch fixes
it by dropping table/db from HMS endpoints so that new table/db succeeds
via Impala shell.
Testing:
1. Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 232 insertions(+), 34 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/13
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#10).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Testing:
1. Added new unit test
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 185 insertions(+), 34 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/10
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@479
PS13, Line 479: .getNextMetastoreEvents(catalog_, currentEventId,
@Vihang: Filter for drop_database in catalogOpExecutor is
List<NotificationEvent> events = getNextMetastoreEventsIfEnabled(eventId,
event -> dbName.equalsIgnoreCase(event.getDbName()) && (
DropDatabaseEvent.DROP_DATABASE_EVENT_TYPE.equals(event.getEventType()) ||
DropTableEvent.DROP_TABLE_EVENT_TYPE.equals(event.getEventType())));
Should it be the same here?
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jul 2021 11:05:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2515
PS10, Line 2515: Table existingTbl = db.getTable(tblName);
: if (existingTbl == null) return null;
> Isn't this same as previous?
It is. We need existingTbl to get eventId which is set in newly initialized incompleteTable.
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@462
PS10, Line 462:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
:
> This code seems to have a race condition, The database could have been adde
Good catch!
We should remove line no. 463 - 467 and let catalogOpExecutor_.removeDbIfNotAddedLater() take care of whether to remove table of not.
Moreover since drop table/db calls in CatalogOpExecutor add entry into deleteEventLog, the same should happen with drop table/db calls here which is missing.
We can do the following:
1. In removeDbIfnotAddedLater() and removeTableIfNotAddedLater(), if the table and db is removed successfully, add drop event id into deleteEntryLog while still holding metastoreDdlLock.
2. CatalogOpExecutor should call removeDbIfNotAddedLater() and removeTableIfNotAddedLater() to drop table/db.
Thoughts ?
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@456
PS13, Line 456: if (!invalidateCacheOnDDLs_) {
: LOG.debug("invalidateCacheOnDDLs_ flag is false, skipping " +
: "cache update for hms operation drop_database on db: " +
: databaseName);
: return;
: }
:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
: MetaStoreUtils.parseDbName(databaseName, serverConf_);
: catName = catAndDbName[0];
: dbName = catAndDbName[1];
: // get DB to drop from catalog cache
: Db dbToDrop = catalog_.getDb(dbName);
: if(dbToDrop == null) {
: // dbToDrop doesn't exist in cache
: return;
: }
: List<NotificationEvent> events;
: try {
: events = MetastoreEventsProcessor
: .getNextMetastoreEvents(catalog_, currentEventId,
: event -> event.getEventType()
: .equalsIgnoreCase(MetastoreEvents.DropDatabaseEvent
: .DROP_DATABASE_EVENT_TYPE)
: && catName.equalsIgnoreCase(event.getCatName())
: && dbName.equalsIgnoreCase(event.getDbName()));
: } catch (ImpalaException e) {
: String errorMsg = "Unable to process Drop database event for db: " +
: dbToDrop.getName();
: LOG.error(errorMsg, e);
: throw new MetaException(errorMsg);
: }
: if (events.isEmpty()) {
: throw new MetaException(
: "Drop database event not received. Check if notification " +
: "events are configured in hive metastore");
: }
: long dropEventId = events.get(events.size() - 1).getEventId();
: boolean isRemoved =
: catalogOpExecutor_.removeDbIfNotAddedLater(dropEventId,
: dbToDrop.getMetaStoreDb());
: if (isRemoved) {
: LOG.info("Removed database: " + databaseName +
: " from cache due to HMS API: drop_database");
> it seems cleaner to refactor this into a separate method just like we have
I intentionally didn't do it since there is only one drop db api. Let me know if you feel otherwise.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2968
PS13, Line 2968: Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
: if (tblProperties == null || MetaStoreUtils.isTransactionalTable(tblProperties)) {
> Can we reuse AcidUtils.isTransactionalTable(catalogTbl.getMetaStoreTable().
How would it help? If we use Impala's AcidUtils method the logic would be:
Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
if (tblProperties == null || AcidUtils.isTransactionalTable(tblProperties)) {
}
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@3003
PS13, Line 3003: if (!invalidateCacheOnDDLs_) {
> When will we set this to false ? invalidateCacheOnDDLs_
By default invalidateCacheOnDDLs_ is set to True. Since it triggers the full reload of the table (after invalidating it from cache), a user can set it to False if there are any performance concerns. The side effect of setting flag to false would be - external tables would be eventually consistent.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jul 2021 12:35:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7381/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 17:30:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 16:48:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9187/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 11
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 19:09:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 14:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9227/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 14
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 13:47:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/8888/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 18:11:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#3).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 403 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/3
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8928/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 22:32:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Aug 2021 18:14:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#15).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in cache\
and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch fixes
it by dropping table/db from HMS endpoints so that new table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 318 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/15
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#5).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 420 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/5
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9146/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Jul 2021 17:55:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7351/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 18:11:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#7).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Testing:
1. Added new unit test
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 163 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/7
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#6).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 370 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/6
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#22).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 326 insertions(+), 48 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/22
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7384/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 11:09:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#20).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 326 insertions(+), 49 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/20
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 20
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#2).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are
accessed from catalog's metastore server.
When perfoming these operations, we acquire lock in MetastoreServiceHandler
to serialize concurrent create/drop operations from catalog HMS endpoints
Testing:
1. Modified existing tests
2. Added new test to access create/drop hms apis in parallel to catch
any potential deadlocks.
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
A fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
5 files changed, 399 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/2
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 2
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2966
PS19, Line 2966: LOG.debug("Invalidating non transactional table {} due to HMS api {}",
Why are you changing metastore to HMS ? Metastore seems to be more generic.
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2348
PS19, Line 2348: eventType.equals(DropDatabaseEvent.DROP_DATABASE_EVENT_TYPE) ||
How about Truncate table ?
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 17:30:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 16: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7364/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 22:47:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 18:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9259/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Aug 2021 17:25:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7369/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Aug 2021 23:20:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9171/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 13:00:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7381/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 23:45:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22:
> This build failed because of the following test
> metadata/test_hms_integration.py::TestHmsIntegrationSanity::test_sanity[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> table_format: text/none] FAILED
>
> The test is unrelated to this patch.
>
> > Patch Set 22: Verified-1
> >
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7384/
Yes, this is a test issue introduced in the recent change IMPALA-8795. I will fix this.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 17:25:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19: Code-Review+1
The patch looks good to me. I see there are some pending comments from Kishen. Will bump it up to +2 once Kishen's is done with the review.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 18:03:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7384/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 17:09:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/9169/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 12:25:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 10:
Ported this commit from internal change: https://github.infra.cloudera.com/CDH/Impala/commit/7420677070851e7e85e541fa613b70e1d681467f
Had to rework on the logic because of Vihang's patch recently merged upstream. After that patch, catalogD handles Create/drop events based on create table/db event id.
Please review this patch and share your feedback.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 12:02:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2515
PS10, Line 2515: Table existingTbl = db.getTable(tblName);
: if (existingTbl == null) return null;
> It is. We need existingTbl to get eventId which is set in newly initialized
Oh I see. I missed that part. Thanks for clarifying.
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@462
PS10, Line 462:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
:
> Good catch!
I agree with the point 2 above. But I think the deleteEventLog should be added from this method not from within removeDbIfnotAddedLater and removeTableIfNotAddedLater since they are used from events processor as well.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@479
PS13, Line 479: .getNextMetastoreEvents(catalog_, currentEventId,
> @Vihang: Filter for drop_database in catalogOpExecutor is
I believe it fetches for DROP_TABLE events as well because a cascade drop operation on db will drop the tables as well and we would need to add them to the deleteEventLog. I don't think this API drop_database supports cascade (although I didn't verify in the code). If this code would be executed in a drop database cascade operation then we should fetch the drop_table events as well.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@456
PS13, Line 456: if (!invalidateCacheOnDDLs_) {
: LOG.debug("invalidateCacheOnDDLs_ flag is false, skipping " +
: "cache update for hms operation drop_database on db: " +
: databaseName);
: return;
: }
:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
: MetaStoreUtils.parseDbName(databaseName, serverConf_);
: catName = catAndDbName[0];
: dbName = catAndDbName[1];
: // get DB to drop from catalog cache
: Db dbToDrop = catalog_.getDb(dbName);
: if(dbToDrop == null) {
: // dbToDrop doesn't exist in cache
: return;
: }
: List<NotificationEvent> events;
: try {
: events = MetastoreEventsProcessor
: .getNextMetastoreEvents(catalog_, currentEventId,
: event -> event.getEventType()
: .equalsIgnoreCase(MetastoreEvents.DropDatabaseEvent
: .DROP_DATABASE_EVENT_TYPE)
: && catName.equalsIgnoreCase(event.getCatName())
: && dbName.equalsIgnoreCase(event.getDbName()));
: } catch (ImpalaException e) {
: String errorMsg = "Unable to process Drop database event for db: " +
: dbToDrop.getName();
: LOG.error(errorMsg, e);
: throw new MetaException(errorMsg);
: }
: if (events.isEmpty()) {
: throw new MetaException(
: "Drop database event not received. Check if notification " +
: "events are configured in hive metastore");
: }
: long dropEventId = events.get(events.size() - 1).getEventId();
: boolean isRemoved =
: catalogOpExecutor_.removeDbIfNotAddedLater(dropEventId,
: dbToDrop.getMetaStoreDb());
: if (isRemoved) {
: LOG.info("Removed database: " + databaseName +
: " from cache due to HMS API: drop_database");
> I intentionally didn't do it since there is only one drop db api. Let me kn
I felt the code readability improves if you refactor it out. But I don't have strong opinion about it.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2968
PS13, Line 2968: Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
: if (tblProperties == null || MetaStoreUtils.isTransactionalTable(tblProperties)) {
> How would it help? If we use Impala's AcidUtils method the logic would be:
I agree that functionally they both for same. My point was more related to consistency within the code. Several other places in the catalogd use AcidUtils.isTransactionalTable. It makes it easier to lookup the places as well easy to change/enhance the implementation if needed. The other advantage is that we don't need to depend on a non-public API in HMS which can be changed/removed without any notice.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Aug 2021 20:43:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#11).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Testing:
1. Added new unit test
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 226 insertions(+), 34 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/11
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 11
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7369/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Aug 2021 17:19:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7347/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 21:46:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7235/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 23:40:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py@775
PS7, Line 775:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py@775
PS7, Line 775: # table already exists
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 12:15:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7232/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 04:43:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@3003
PS13, Line 3003: if (!invalidateCacheOnDDLs_) {
When will we set this to false ? invalidateCacheOnDDLs_
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jul 2021 01:15:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jul 2021 11:47:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jul 2021 12:29:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7347/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jul 2021 15:48:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2515
PS10, Line 2515: Table existingTbl = db.getTable(tblName);
: if (existingTbl == null) return null;
Isn't this same as previous?
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2519
PS10, Line 2519: incompleteTable.setCreateEventId(existingTbl.getCreateEventId());
> @Vihang: This was probably missed in HANDLE create/drop patch. Please take
Yeah, that looks correct to me. Thanks for fixing it.
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@462
PS10, Line 462:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
:
This code seems to have a race condition, The database could have been added after line 466 is executed.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@456
PS13, Line 456: if (!invalidateCacheOnDDLs_) {
: LOG.debug("invalidateCacheOnDDLs_ flag is false, skipping " +
: "cache update for hms operation drop_database on db: " +
: databaseName);
: return;
: }
:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
: MetaStoreUtils.parseDbName(databaseName, serverConf_);
: catName = catAndDbName[0];
: dbName = catAndDbName[1];
: // get DB to drop from catalog cache
: Db dbToDrop = catalog_.getDb(dbName);
: if(dbToDrop == null) {
: // dbToDrop doesn't exist in cache
: return;
: }
: List<NotificationEvent> events;
: try {
: events = MetastoreEventsProcessor
: .getNextMetastoreEvents(catalog_, currentEventId,
: event -> event.getEventType()
: .equalsIgnoreCase(MetastoreEvents.DropDatabaseEvent
: .DROP_DATABASE_EVENT_TYPE)
: && catName.equalsIgnoreCase(event.getCatName())
: && dbName.equalsIgnoreCase(event.getDbName()));
: } catch (ImpalaException e) {
: String errorMsg = "Unable to process Drop database event for db: " +
: dbToDrop.getName();
: LOG.error(errorMsg, e);
: throw new MetaException(errorMsg);
: }
: if (events.isEmpty()) {
: throw new MetaException(
: "Drop database event not received. Check if notification " +
: "events are configured in hive metastore");
: }
: long dropEventId = events.get(events.size() - 1).getEventId();
: boolean isRemoved =
: catalogOpExecutor_.removeDbIfNotAddedLater(dropEventId,
: dbToDrop.getMetaStoreDb());
: if (isRemoved) {
: LOG.info("Removed database: " + databaseName +
: " from cache due to HMS API: drop_database");
it seems cleaner to refactor this into a separate method just like we have removeTableIfExists() below.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2931
PS13, Line 2931: invalidateNonTransactionalTableIfExists
what is the functional difference between this and renameNonTransactionalTableIfExists? Can we consolidate the two if there is an overlap.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2968
PS13, Line 2968: Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
: if (tblProperties == null || MetaStoreUtils.isTransactionalTable(tblProperties)) {
Can we reuse AcidUtils.isTransactionalTable(catalogTbl.getMetaStoreTable().getParameters()) method here?
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jul 2021 19:46:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 16:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7364/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 16:34:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has removed a vote on this change.
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#8).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Testing:
1. Added new unit test
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
4 files changed, 163 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/8
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7235/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jun 2021 17:44:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
> Patch Set 17: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7368/
The failed test in the build is tests/custom_cluster/test_event_processing.py::TestEventProcessing::test_transactional_insert_events
Ran it locally and it is passing. The failure seems to intermittent issue.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Aug 2021 11:04:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22: Verified+1 Code-Review+2
> > This build failed because of the following test
> > metadata/test_hms_integration.py::TestHmsIntegrationSanity::test_sanity[protocol:
> > beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> > 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> > table_format: text/none] FAILED
> >
> > The test is unrelated to this patch.
> >
> > > Patch Set 22: Verified-1
> > >
> > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7384/
>
> Yes, this is a test issue introduced in the recent change
> IMPALA-8795. I will fix this.
I am going to give a +1 verified since the only test which failed is a known to be unrelated to this patch. I have submitted https://gerrit.cloudera.org/#/c/17767/ to fix the test.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 18:11:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2966
PS19, Line 2966: LOG.debug("Invalidating non transactional table {} due to HMS api {}",
> Why are you changing metastore to HMS ? Metastore seems to be more generic.
There was no specific reason. Will revert back to using Metastore
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17576/19/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2348
PS19, Line 2348: eventType.equals(DropDatabaseEvent.DROP_DATABASE_EVENT_TYPE) ||
> How about Truncate table ?
Alter_Table event is generated for truncate_table. As of now, we just invalidate the table and not add it to deleteEventLog
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 19:38:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
kishen@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 17:57:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7232/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jun 2021 22:44:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17576/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@467
PS1, Line 467: Db removedDb = catalog_.removeDbIfExists(dbToDrop.getMetaStoreDb(), dbFound, dbMatched);
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/17576/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@3020
PS1, Line 3020: String dbName, String tblName, String apiName) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/17576/1/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java
File fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java:
http://gerrit.cloudera.org:8080/#/c/17576/1/fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsParallelCreateDropTest.java@46
PS1, Line 46: "thrift://localhost:" + ((CatalogServiceTestCatalog.CatalogServiceTestHMSCatalog) catalog_).getPort());
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/17576/1/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/17576/1/tests/custom_cluster/test_metastore_service.py@729
PS1, Line 729: p
flake8: F841 local variable 'prev_get_table_response' is assigned to but never used
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Jun 2021 17:35:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7368/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Aug 2021 16:16:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Reviewed-on: http://gerrit.cloudera.org:8080/17576
Reviewed-by: <ki...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 326 insertions(+), 48 deletions(-)
Approvals:
kishen@cloudera.com: Looks good to me, but someone else must approve
Vihang Karajgaonkar: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9228/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 15:34:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#18).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 323 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/18
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17576/6/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/6/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@485
PS6, Line 485: .equalsIgnoreCase(MetastoreEvents.DropDatabaseEvent.DROP_DATABASE_EVENT_TYPE)
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/17576/6/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:
http://gerrit.cloudera.org:8080/#/c/17576/6/tests/custom_cluster/test_metastore_service.py@773
PS6, Line 773: #
flake8: E265 block comment should start with '# '
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Jul 2021 17:44:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 19:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9272/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 17:17:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 22:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/9280/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Aug 2021 11:33:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/10/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@462
PS10, Line 462:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
:
> I agree with the point 2 above. But I think the deleteEventLog should be ad
Sure.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@479
PS13, Line 479: .getNextMetastoreEvents(catalog_, currentEventId,
> I believe it fetches for DROP_TABLE events as well because a cascade drop o
You are right. Cascade for drop_database is supported (and implemented) in HiveMetastoreClient but it is not part of thrift interface. We don't need to check for drop_table events here.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@456
PS13, Line 456: if (!invalidateCacheOnDDLs_) {
: LOG.debug("invalidateCacheOnDDLs_ flag is false, skipping " +
: "cache update for hms operation drop_database on db: " +
: databaseName);
: return;
: }
:
: // Parse db name. Throw error if parsing fails.
: String dbName;
: String catName;
: String[] catAndDbName =
: MetaStoreUtils.parseDbName(databaseName, serverConf_);
: catName = catAndDbName[0];
: dbName = catAndDbName[1];
: // get DB to drop from catalog cache
: Db dbToDrop = catalog_.getDb(dbName);
: if(dbToDrop == null) {
: // dbToDrop doesn't exist in cache
: return;
: }
: List<NotificationEvent> events;
: try {
: events = MetastoreEventsProcessor
: .getNextMetastoreEvents(catalog_, currentEventId,
: event -> event.getEventType()
: .equalsIgnoreCase(MetastoreEvents.DropDatabaseEvent
: .DROP_DATABASE_EVENT_TYPE)
: && catName.equalsIgnoreCase(event.getCatName())
: && dbName.equalsIgnoreCase(event.getDbName()));
: } catch (ImpalaException e) {
: String errorMsg = "Unable to process Drop database event for db: " +
: dbToDrop.getName();
: LOG.error(errorMsg, e);
: throw new MetaException(errorMsg);
: }
: if (events.isEmpty()) {
: throw new MetaException(
: "Drop database event not received. Check if notification " +
: "events are configured in hive metastore");
: }
: long dropEventId = events.get(events.size() - 1).getEventId();
: boolean isRemoved =
: catalogOpExecutor_.removeDbIfNotAddedLater(dropEventId,
: dbToDrop.getMetaStoreDb());
: if (isRemoved) {
: LOG.info("Removed database: " + databaseName +
: " from cache due to HMS API: drop_database");
> I felt the code readability improves if you refactor it out. But I don't ha
Sure. I will move it to a different method.
http://gerrit.cloudera.org:8080/#/c/17576/13/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2968
PS13, Line 2968: Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
: if (tblProperties == null || MetaStoreUtils.isTransactionalTable(tblProperties)) {
> I agree that functionally they both for same. My point was more related to
Got it. Will switch to using AcidUtils
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Aug 2021 10:56:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7368/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Aug 2021 10:00:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#14).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
Consider a scenario where a table/db already exists and a user drops it via
catalog metastore server endpoint. When recreating the same via Impala
Shell, user gets an error that table/db already exists. This patch fixes
it by dropping table/db from HMS endpoints so that new table/db succeeds
via Impala shell.
Testing:
1. Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 317 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/14
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 14
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7375/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Aug 2021 12:01:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 12
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jul 2021 01:02:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7356/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jul 2021 06:22:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 18:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17576/17/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17576/17/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2967
PS17, Line 2967: ame);
: org.apache.impala.cat
> nit, can go in the same line above. I would suggest you to have your IDE se
Thanks for sharing the style link. The formatting got messed up in the last conflict resolution. Will fix it.
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@861
PS18, Line 861: msDb
> pls update the java doc
Sorry, I missed it. Thanks for pointing it out.
http://gerrit.cloudera.org:8080/#/c/17576/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@865
PS18, Line 865: String dbName
> this can go into the above line now.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Aug 2021 16:54:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, kishen@cloudera.com, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17576
to look at the new patch set (#19).
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS
apis are accessed from catalog's metastore server.
This patch fixes a scenario where if table/db already exists in
cache and a user drops it via catalog metastore server endpoint
(i.e drop_table HMS api). When recreating the same via Impala
Shell, user gets an error that table/db already exists. The patch
fixes it by dropping table/db from HMS endpoints so that new
table/db succeeds
Testing:
Added new unit tests which cover drop_database and
drop_table HMS API
Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_metastore_service.py
M tests/util/event_processor_utils.py
6 files changed, 322 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/19
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 )
Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
......................................................................
Patch Set 16:
The following test is failing in the latest run
19:12:11 [ERROR] Tests run: 117, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 143.257 s <<< FAILURE! - in org.apache.impala.planner.PlannerTest
19:12:11 [ERROR] testHbaseNoKeyEstimate(org.apache.impala.planner.PlannerTest) Time elapsed: 0.107 s <<< FAILURE!
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:930)
at org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:845)
at org.apache.impala.planner.PlannerTest.testHbaseNoKeyEstimate(PlannerTest.java:234)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
The failure seems to be unrelated to this patch.
> Patch Set 16: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7364/
--
To view, visit http://gerrit.cloudera.org:8080/17576
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c
Gerrit-Change-Number: 17576
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <ki...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Aug 2021 09:44:31 +0000
Gerrit-HasComments: No