[jira] [Resolved] (IMPALA-12460) Add lag and histogram of event processing in the log

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang resolved IMPALA-12460.
-------------------------------------
    Fix Version/s: Impala 4.4.0
       Resolution: Fixed

> Add lag and histogram of event processing in the log
> ----------------------------------------------------
>
>                 Key: IMPALA-12460
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12460
>             Project: IMPALA
>          Issue Type: Improvement
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>             Fix For: Impala 4.4.0
>
>
> At the end of processing each event batch, we log the processing time of that batch and the latest event info inside HMS, e.g.
> {code:java}
> I0923 04:02:05.530905 12886 MetastoreEventsProcessor.java:859] Time elapsed in processing event batch: 28m1s
> I0923 04:02:05.539295 12886 MetastoreEventsProcessor.java:730] Latest event in HMS: id=56918528, time=1695459725{code}
> We should also log the lag of the event processing which is (latestEventTime - lastSyncedEventTime). For each event batch that is slow to synced, we should also print a histogram to show what are the top10 expensive events to sync, and what are the top10 tables that take longest time to sync. So admins can decide which tables to disable event processing on them.
> CC [~hemanth619] , [~csringhofer] , [~wzhou] , [~kdeschle] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org