You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Sarath Subramanian (Jira)" <ji...@apache.org> on 2021/06/07 22:11:00 UTC
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on
metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sarath Subramanian updated ATLAS-4330:
--------------------------------------
Description:
In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
{code}
was:
In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas:
!image-2021-06-07-15-03-28-166.png!
> Add Kafka topics lag information on metrics and log
> ---------------------------------------------------
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
> Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so).
>
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
>
>
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)