[jira] [Created] (IMPALA-10926) Sync db/table to latest event id for ddl operations in catalog megastore server

["Sourabh Goyal (Jira)" <ji...@apache.org>]

Sourabh Goyal created IMPALA-10926:
--------------------------------------

             Summary: Sync db/table to latest event id for ddl operations in catalog megastore server
                 Key: IMPALA-10926
                 URL: https://issues.apache.org/jira/browse/IMPALA-10926
             Project: IMPALA
          Issue Type: Task
            Reporter: Sourabh Goyal
            Assignee: Sourabh Goyal


Whenever DDL operations like add/remove partitions, alter table etc are performed from catalog HMS endpoints, we will do the following: 
 # Acquire lock on db/table 
 # Perform HMS operation
 # For db/table in step 1, fetch all events from HMS
 # Apply those events on the db/table in catalogd cache 
 # Set the last processed event id for db/table 

In addition to the above, the following needs to be handled as well: 
 # Event processor should skip processing an event on a db/table if the db/table is already synced till that event id. 
 # If last synced event id in table/db is less than the event being processed by event processor, it should set the last synced db/table event id after successfully processing the event. 
 # Full table refresh should set the last event processed. 

cc - [~kishendas] [~vihangk1]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)