You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by "Jasper Knulst (Jira)" <ji...@apache.org> on 2022/05/11 15:47:00 UTC

[jira] [Created] (ATLAS-4602) Report Lag for consuming from ATLAS_HOOK

Jasper Knulst created ATLAS-4602:
------------------------------------

             Summary: Report Lag for consuming from ATLAS_HOOK
                 Key: ATLAS-4602
                 URL: https://issues.apache.org/jira/browse/ATLAS-4602
             Project: Atlas
          Issue Type: Improvement
          Components:  atlas-core
    Affects Versions: 2.2.0
            Reporter: Jasper Knulst
             Fix For: trunk
         Attachments: image-2022-05-11-17-42-12-250.png

Currently the 'Stats' webUI function shows some details about the consumption from the ATLAS_HOOK Kafka topic where changes from Hive Metastore arrive.

 !image-2022-05-11-17-42-12-250.png! 

By far the most important metric is not available though; the lag the atlas server consumer-group has in consuming Hive updates.

Monitoring the lag is very important as trust in Atlas is greatly undermined when changes are not reflected in Atlas within seconds. I have had numerous occasions where ATLAS_HOOK consumption was slowing down silently and atlas was behind tens of thousands (or 2 days) worth of messages.

There should be a new metric for the lag on the stats page to quickly identify a possible reason for slow Atlas updates




--
This message was sent by Atlassian Jira
(v8.20.7#820007)