[Impala-ASF-CR] IMPALA-12832: Implicit invalidate metadata on event failures

["Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>]

Sai Hemanth Gantasala has uploaded a new patch set (#18) to the change originally created by k.venureddy2103@gmail.com. ( http://gerrit.cloudera.org:8080/21065 )

Change subject: IMPALA-12832: Implicit invalidate metadata on event failures
......................................................................

IMPALA-12832: Implicit invalidate metadata on event failures

At present, failure in event processing needs manual invalidate
metadata. This patch implicitly invalidates the table upon failures
in processing of table events with new 'process_event_failure' flag.
And a new 'auto_global_invalidate_metadata' flag is added to global
invalidate metadata automatically when event processor goes to
non-active state.

Note: Also introduced a config
'inject_process_event_failure_event_types' for automated tests to
simulate event processor failures. This config is used to specify what
event types can be intentionally failed. This config should only be
used for testing purpose. Need IMPALA-12851 as a prerequisite

Testing:
- Added end-to-end tests to mimic failures in event processor and verified
that event processor is active
- Added unit test to verify the 'auto_global_invalidate_metadata' config
- Passed FE tests

Co-Authored-by: Sai Hemanth Gantasala <sa...@cloudera.com>

Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/common/impala_test_suite.py
A tests/metadata/__init__.py
M tests/metadata/test_event_processing.py
A tests/metadata/test_event_processing_base.py
A tests/metadata/test_event_processing_error.py
M tests/util/event_processor_utils.py
16 files changed, 1,023 insertions(+), 305 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/21065/18
-- 
To view, visit http://gerrit.cloudera.org:8080/21065
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia67fc04c995802d3b6b56f79564bf0954b012c6c
Gerrit-Change-Number: 21065
Gerrit-PatchSet: 18
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>