[impala] branch master updated: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 66484a4c0 IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables
66484a4c0 is described below

commit 66484a4c081f3242750a3a0e04159dd4580b37a4
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Tue Sep 20 10:41:24 2022 +0800

    IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables
    
    When applying ALTER_PARTITION events on transactional tables, we refresh
    the partition using the metadata in events if
    hms_event_incremental_refresh_transactional_table is enabled (which is
    the default). This could be wrong if the ALTER_PARTITION event is stale.
    The partition metadata will be rolled back to a stale state.
    
    This patch compares the eventId with the createEventId of the table and
    ignores those ALTER_PARTITION events that have older (smaller) event
    ids. Note that we already do this for many other event types,
    ALTER_PARTITION is somehow missing the checks.
    
    Eventually we should depend on the lastSyncedEventId and replace
    createEventId with it. The self-event detection can also be replaced
    since self-events are also stale events. These will be addressed in
    IMPALA-10976.
    
    Tests
    - Verified locally with local-catalog mode and event-processor enabled
      and iterated test_acid_compute_stats for 1400 times. Without the fix,
      the test would fail in tens of runs.
    
    Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a
    Reviewed-on: http://gerrit.cloudera.org:8080/19020
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 7e8f866e0..1be372d36 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -4453,6 +4453,11 @@ public class CatalogOpExecutor {
     if (!(table instanceof HdfsTable)) {
       throw new CatalogException("Partition event received on a non-hdfs table");
     }
+    if (eventId > 0 && eventId <= table.getCreateEventId()) {
+      LOG.debug("Not reloading partitions of table {}.{} for event {} since it is " +
+          "recreated at event {}.", dbName, tblName, eventId, table.getCreateEventId());
+      return 0;
+    }
     try {
       tryWriteLock(table, reason);
       long newCatalogVersion = catalog_.incrementAndGetCatalogVersion();