[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

["Madhan Neethiraj (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Madhan Neethiraj updated ATLAS-4330:
------------------------------------
    Fix Version/s:     (was: 2.3.0)

> Add Kafka topics lag information on metrics and log
> ---------------------------------------------------
>
>                 Key: ATLAS-4330
>                 URL: https://issues.apache.org/jira/browse/ATLAS-4330
>             Project: Atlas
>          Issue Type: Improvement
>          Components:  atlas-core
>    Affects Versions: 2.1.0
>            Reporter: Sarath Subramanian
>            Assignee: Sarath Subramanian
>            Priority: Major
>              Labels: kafka, metrics, topic-lag
>             Fix For: 3.0.0
>
>         Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backed up messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)