[jira] [Resolved] (HDDS-17038) OzoneManagerDoubleBuffer metrics disappear permanently after a follower installs a Ratis snapshot

["Chi-Hsuan Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chi-Hsuan Huang resolved HDDS-17038.
------------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> OzoneManagerDoubleBuffer metrics disappear permanently after a follower installs a Ratis snapshot
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17038
>                 URL: https://issues.apache.org/jira/browse/HDDS-17038
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: OM HA
>    Affects Versions: 2.0.0, 2.2.0, 2.2.1
>            Reporter: Aleksei Ieshin
>            Assignee: Andy Huang
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> After an OM installs a Ratis snapshot (for example, a lagging follower catching up), all {{OzoneManagerDoubleBufferMetrics}} ({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}}, {{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published. They stay missing until the OM JVM restarts. The double buffer itself keeps flushing; only the metrics source is gone.
> Cause:
>  # {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls {{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}} calls {{{}metrics.unRegister(){}}}, which runs {{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
>  # {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer via {{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls {{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
>  # {{create()}} is a lazy singleton:
>  
> !http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone! 
> {code:java}
> public static synchronized OzoneManagerDoubleBufferMetrics create() {
>   if (instance != null) {
>     return instance;      // still the old object, whose source was unregistered in step 1
>   } ... {code}
> {{`unRegister()`}} never resets {{{}instance{}}}, so the new double buffer updates an object that no longer belongs to the metrics system, and nothing is exported.
> Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a 3-node OM HA cluster, two followers lost these metrics after snapshot installs (one during a RATIS-2240 snapshot-install deadlock, one during a period of heavy load). Both got them back only after their pods restarted. The leader, which never installs a snapshot, was unaffected. This silently blinds dashboards and alerts built on double-buffer flush time.
> *Proposed fix:* make {{unRegister()}} reset the singleton, so the next {{create()}} registers a fresh source:
>  
> {code:java}
> public static synchronized void unRegister() {   // or keep it an instance method, but clear the static
>   DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
>   instance = null;
> } {code}
> Alternatively, {{create()}} could re-register when {{instance != null}} but the source is missing from the metrics system.
> *Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via {{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}. Assert that {{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}} is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a further write. Possibly related: HDDS-16908 (intermittent failure in {{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since tests share the static singleton.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org