You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2021/06/04 12:20:50 UTC
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17542
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Event processing can fail when it needs to process multiple events
belonging to the same table, and if one of the events is a DROP
TABLE.
E.g. imagine the following scenario:
Hive> alter table foo set location '/tmp';
Hive> drop table foo;
If the two DDLs are executed quickly there's good chance that Impala's
event processor will handle these in one event batch. In this case
during the processing of ALTER TABLE Impala will try to load table
'foo' which will fail (because Hive already deleted the table).
The event processor is not prepared for this exception and gets into
a broken state which means no further events getting processed.
This patch adds exception handling to
CatalogServiceCatalog.reloadTableIfExists() for the case when the
table cannot be loaded, in which case it returns false.
Testing:
* added e2e test
Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_event_processing.py
2 files changed, 43 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17542/1
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 15:41:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 09:40:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17542
to look at the new patch set (#3).
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Event processing can fail when it needs to process multiple events
belonging to the same table, and if one of the events is a DROP
TABLE.
E.g. imagine the following scenario:
Hive> alter table foo set location '/tmp';
Hive> drop table foo;
If the two DDLs are executed quickly there's good chance that Impala's
event processor will handle these in one event batch. In this case
during the processing of ALTER TABLE Impala will try to load table
'foo' which will fail (because Hive already deleted the table).
The event processor is not prepared for this exception and gets into
a broken state which means no further events getting processed.
This patch adds exception handling to
CatalogServiceCatalog.reloadTableIfExists() for the case when the
table cannot be loaded, in which case it returns false.
Testing:
* added e2e test
Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_event_processing.py
3 files changed, 40 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17542/3
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2740
PS1, Line 2740: reloadTableIfExists
> I think the problem is not limited to tables but rather create, drop events
I discussed with Zoltan offline and we agreed that since https://gerrit.cloudera.org/#/c/17308/ is fairly complex and and harder to backport it would be good to get this patch in.
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py@220
PS1, Line 220: tmp
Setting table directory to /tmp can lead to flakiness if the drop table statements below removes contents of /tmp. Can you set it to a dir within /tmp instead and change the table to external?
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py@224
PS1, Line 224: tbl_name = "bar"
: self.run_stmt_in_hive("""
: drop table if exists {db}.{tbl};
: create table {db}.{tbl} (id int);
: insert into {db}.{tbl} values(2);""".format(db=db_name, tbl=tbl_name))
: EventProcessorUtils.wait_for_event_processing(self, event_proc_timeout)
: data = self.execute_scalar("select * from %s.%s" % (db_name, tbl_name))
: assert data == '2'
Or you can simply do the following
assert EventProcessorUtils.get_event_processor_status() == "ACTIVE"
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Jun 2021 18:52:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 5: Code-Review+2
Carry +2
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 09:39:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 10:02:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17542
to look at the new patch set (#2).
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Event processing can fail when it needs to process multiple events
belonging to the same table, and if one of the events is a DROP
TABLE.
E.g. imagine the following scenario:
Hive> alter table foo set location '/tmp';
Hive> drop table foo;
If the two DDLs are executed quickly there's good chance that Impala's
event processor will handle these in one event batch. In this case
during the processing of ALTER TABLE Impala will try to load table
'foo' which will fail (because Hive already deleted the table).
The event processor is not prepared for this exception and gets into
a broken state which means no further events getting processed.
This patch adds exception handling to
CatalogServiceCatalog.reloadTableIfExists() for the case when the
table cannot be loaded, in which case it returns false.
Testing:
* added e2e test
Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_event_processing.py
2 files changed, 36 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17542/2
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Event processing can fail when it needs to process multiple events
belonging to the same table, and if one of the events is a DROP
TABLE.
E.g. imagine the following scenario:
Hive> alter table foo set location '/tmp';
Hive> drop table foo;
If the two DDLs are executed quickly there's good chance that Impala's
event processor will handle these in one event batch. In this case
during the processing of ALTER TABLE Impala will try to load table
'foo' which will fail (because Hive already deleted the table).
The event processor is not prepared for this exception and gets into
a broken state which means no further events getting processed.
This patch adds exception handling to
CatalogServiceCatalog.reloadTableIfExists() for the case when the
table cannot be loaded, in which case it returns false.
Testing:
* added e2e test
Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Reviewed-on: http://gerrit.cloudera.org:8080/17542
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_event_processing.py
3 files changed, 42 insertions(+), 16 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7205/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 18:12:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8846/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Jun 2021 12:42:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7209/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 09:40:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17542
to look at the new patch set (#5).
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Event processing can fail when it needs to process multiple events
belonging to the same table, and if one of the events is a DROP
TABLE.
E.g. imagine the following scenario:
Hive> alter table foo set location '/tmp';
Hive> drop table foo;
If the two DDLs are executed quickly there's good chance that Impala's
event processor will handle these in one event batch. In this case
during the processing of ALTER TABLE Impala will try to load table
'foo' which will fail (because Hive already deleted the table).
The event processor is not prepared for this exception and gets into
a broken state which means no further events getting processed.
This patch adds exception handling to
CatalogServiceCatalog.reloadTableIfExists() for the case when the
table cannot be loaded, in which case it returns false.
Testing:
* added e2e test
Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_event_processing.py
3 files changed, 42 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17542/5
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7205/
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 00:07:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 18:12:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17542/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17542/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2738
PS2, Line 2738: DatabaseNotFoundException
Wouldn't it be more intuitive if both exceptions were handled at the same level? This functions is called only here: https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L563
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Jun 2021 15:59:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 3:
Thank you all!
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 18:12:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 3:
(1 comment)
Thanks for the comment!
http://gerrit.cloudera.org:8080/#/c/17542/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17542/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2738
PS2, Line 2738:
> Wouldn't it be more intuitive if both exceptions were handled at the same l
Done
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 09:41:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2740
PS1, Line 2740: reloadTableIfExists
I think the problem is not limited to tables but rather create, drop events on tables, partitions and databases as well. I have a patch which is in review which handles these cases all together instead of handling them one by one. I am hoping to update it today or by Monday so that we can merge it soon. https://gerrit.cloudera.org/#/c/17308/
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Jun 2021 16:56:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 10:01:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8851/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Jun 2021 09:14:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 3: Code-Review+2
+1 and carrying forward Csaba's +1
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 17:59:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Event processing fails on multiple events + DROP TABLE
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17542 )
Change subject: IMPALA-10187: Event processing fails on multiple events + DROP TABLE
......................................................................
Patch Set 2:
(3 comments)
Thanks for the comments, Vihang!
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17542/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2740
PS1, Line 2740: reloadTableIfExists
> I discussed with Zoltan offline and we agreed that since https://gerrit.clo
Done
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py
File tests/custom_cluster/test_event_processing.py:
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py@220
PS1, Line 220: es
> Setting table directory to /tmp can lead to flakiness if the drop table sta
Thanks for pointing this out. I've switched to use ALTER TABLE SET TBLPROPERTIES which might be the safest thing to to.
http://gerrit.cloudera.org:8080/#/c/17542/1/tests/custom_cluster/test_event_processing.py@224
PS1, Line 224: assert EventProcessorUtils.get_event_processor_status() == "ACTIVE"
:
: def __run_event_based_replication_tests(self, transactional=True):
: """Hive Replication relies on the insert events generated on the tables.
: This test issues some basic replication commands from Hive and makes sure
: that the replicated table has correct data."""
: TBLPROPERTIES = self.__get_transactional_tblproperties(transactional)
: source_db = self.__g
> Or you can simply do the following
Ah, nice. Done.
--
To view, visit http://gerrit.cloudera.org:8080/17542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2bd72247293d830063f66eecf9e4ba69a439a605
Gerrit-Change-Number: 17542
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Jun 2021 08:55:10 +0000
Gerrit-HasComments: Yes