[Impala-ASF-CR] IMPALA-10502: Handle CREATE/DROP events correctly

["Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>]

Vihang Karajgaonkar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/17308 )

Change subject: IMPALA-10502: Handle CREATE/DROP events correctly
......................................................................

IMPALA-10502: Handle CREATE/DROP events correctly

The current way to detect self-events in case of CREATE/DROP events on
database, table and partition is problematic when the same object is
created and dropped repeatedly in quick succession. This happens mainly
due to couple of reasons. For example if we have a sequence of
CREATE_TABLE, DROP_TABLE, CREATE_TABLE ... on the same table, it is
possible that when the create table event is being processed, the table
is not present in catalog because it was dropped recently. In such a
case, events processor does not have enough state information in
catalogd to determine that this table has been dropped from the
catalogd and the event should be ignored. Similarly, if a drop event
is being processed, it is possible that the table has been recreated
with the same name when the drop event is received. In such a case,
events processor removes the table from the catalogd.

This can cause problems for queries which expect the table to exist or
not exist. E.g create table query fails with a table already exists or
a drop table query fails with table does not exist error.

In order to fix this issue, catalogd now keeps track of dropped objects
in a deleteLog which are garbage collected as the events come in. Every
time a database, table or parition is dropped, the deleteLog is
populated with the the drop event id generated due to the drop
operation. This deleteLog is looked up when the event is received to
determine if the event can be ignored.

Testing:
1. Added a new test which loops to create create/drop events for
database, table and partitions.
2. Added new metrics which the test verifies to ensure that events
don't create or drop the object.

Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java
A fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java
A fe/src/main/java/org/apache/impala/catalog/events/EventFactory.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServer.java
M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
A fe/src/test/java/org/apache/impala/catalog/metastore/AbstractCatalogMetastoreTest.java
M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsFileMetadataTest.java
M fe/src/test/java/org/apache/impala/catalog/metastore/EnableCatalogdHmsCacheFlagTest.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
A fe/src/test/java/org/apache/impala/testutil/CatalogTestMetastoreServer.java
M tests/custom_cluster/test_event_processing.py
29 files changed, 2,579 insertions(+), 1,151 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/17308/10
-- 
To view, visit http://gerrit.cloudera.org:8080/17308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a
Gerrit-Change-Number: 17308
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>