You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Vinish Reddy (Jira)" <ji...@apache.org> on 2022/02/07 17:09:00 UTC
[jira] [Updated] (HUDI-3373) Add zero value counter metrics because of Prometheus PushGateway stale metrics problem
[ https://issues.apache.org/jira/browse/HUDI-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinish Reddy updated HUDI-3373:
-------------------------------
Component/s: metrics
> Add zero value counter metrics because of Prometheus PushGateway stale metrics problem
> --------------------------------------------------------------------------------------
>
> Key: HUDI-3373
> URL: https://issues.apache.org/jira/browse/HUDI-3373
> Project: Apache Hudi
> Issue Type: Task
> Components: metrics
> Reporter: Vinish Reddy
> Priority: Minor
>
> Prometheus [Pushgateway|https://prometheus.io/docs/practices/pushing/] never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted via the Pushgateway's API. When deltastreamer finds that there are no new records to be read from the source since the last checkpoint it doesn't emit a metric because the write path is not triggered and leads to a graph like below in case of PushGateway metrics reporter. For other metrics reporter it would be a null value and no data will be shown in the graph.
> {{22/02/07 11:23:10 INFO DeltaSync: No new data, source checkpoint has not changed. Nothing to commit. Old checkpoint=(Option\{val=20220207062847034}). New Checkpoint=(20220207062847034)}}
> {{!https://user-images.githubusercontent.com/16958856/152831831-7c4baab1-8b01-4100-8b64-40ec702e9749.png!}}
>
> To give a better representation of the actual metrics for Prometheus PushGateway, the counter metrics have to be updated with zero value so that it's better than having stale data which was ingested as part of the last checkpoint.
> !https://user-images.githubusercontent.com/16958856/152831933-7238666b-b6b4-4ac6-9962-399b6a4cf81b.png!
>
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)