[jira] [Resolved] (IMPALA-11490) More metrics to debug event processing lagging behind

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-11490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang resolved IMPALA-11490.
-------------------------------------
    Target Version: Impala 4.2.0
        Resolution: Fixed

> More metrics to debug event processing lagging behind
> -----------------------------------------------------
>
>                 Key: IMPALA-11490
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11490
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>              Labels: supportability
>
> Event processor could lag behind in many cases, e.g. processing lots events on large tables, waiting for table locks held by manual refresh or other metadata operations, etc.
> Currently we have metric on the last synced event id. We should also add metric on the latest event id in HMS. Users can compare them to know whether event processing is lagging behind.
> We should also add logs/metrics on tables that take long time in event processing, especially those longer than the event polling interval. So users can decide whether to disable event processing on them, or reduce concurrency of metadata operations on them.
> Some metrics like average events processing duration in the last 5min, 30min or 1h will also be helpful.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)