[Impala-ASF-CR] IMPALA-9101: Add support for detecting self-events on partition events

["Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>]

Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14799 )

Change subject: IMPALA-9101: Add support for detecting self-events on partition events
......................................................................

IMPALA-9101: Add support for detecting self-events on partition events

This commit redoes some of the self-event detection logic, specifically for the partition
events. Before the patch, the self-event identifiers for a partition were stored at a
table level when generating the partition events. This was problematic since unlike
ADD_PARTITION and DROP_PARTITION event, ALTER_PARTITION event is generated one per
partition. Due to this if there are multiple ALTER_PARTITION events generated, only the
first event is identified as a self-event and the reset of the events are processed. This
patch fixes this by adding the self-event identifiers to each partition so that when the
event is later received, each ALTER_PARTITION uses the state stored in HdfsPartition to
evaluate the self-events. The patch makes sure that the event processor takes a table lock
during self-event evaluation to avoid races with other parts of the code which try to
modify the table at the same time.

Additionally, this patch also changes the event processor to refresh a loaded table
(incomplete tables are not refreshed) when a ALTER_TABLE event is received instead of
invalidating the table. This makes the events processor consistent with respect to all the
other event types. In future, we should add a flag to choose the behavior preference
(prefer invalidate or refresh).

Also, this patch fixes the following related issues:
1. Self-event logic was not triggered for alter database events when user modifies the
comment on the database.
2. In case of queries like "alter table add if not exists partition...", the partition is
not added since its pre-existing. The self-event identifiers should not be added in such
cases since no event is expected from such queries.
3. Changed wait_for_event_processing test util method in EventProcessorUtils to use a more
deterministic way to determine if the catalog updates have propogated to impalad instead
of waiting for a random duration of time. This also speeds up the event processing tests
significantly.

Testing Done:
1. Added a e2e self-events test which runs multiple impala queries and makes sure that the
event is skips processing.
2. Ran MetastoreEventsProcessorTest
3. [TODO] Run core tests on CDH and CDP builds.

Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_event_processing.py
M tests/util/event_processor_utils.py
13 files changed, 1,032 insertions(+), 643 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/14799/3
-- 
To view, visit http://gerrit.cloudera.org:8080/14799
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9b4148f6be0f9f946c8ad8f314d64b095731744c
Gerrit-Change-Number: 14799
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>