[jira] [Resolved] (IMPALA-13801) Enhance Lag mechanism in hierarchical metastore event processing

["Venugopal Reddy K (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Venugopal Reddy K resolved IMPALA-13801.
----------------------------------------
    Resolution: Fixed

> Enhance Lag mechanism in hierarchical metastore event processing
> ----------------------------------------------------------------
>
>                 Key: IMPALA-13801
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13801
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Venugopal Reddy K
>            Assignee: Venugopal Reddy K
>            Priority: Critical
>              Labels: catalog-2024
>
> 1. Need to redefine the lag in the hierarchical processing mode. Hierarchical mode currently have a mechanism to check the total number of outstanding events at the moment so that memory usage pressure on catalogd can be controlled when event processing becomes slow i.e., iff outstanding event count exceeds max_outstanding_events_on_executors configured value.
> 2. Currently, with enable_hierarchical_event_processing as true(introduced with https://issues.apache.org/jira/browse/IMPALA-12709), lastSyncedEventId_ and lastSyncedEventTimeSecs_ are updated upon event dispatch to EventExecutorService for processing on respective DbEventExecutor and/or TableEventExecutor. So lastSyncedEventId_ and lastSyncedEventTimeSecs_ doesn't actually mean events are processed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)