[jira] [Commented] (HBASE-28123) TableRequests metrics broken after altering table

["Xiao Liu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-28123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063881#comment-18063881 ] 

Xiao Liu commented on HBASE-28123:
----------------------------------

Hi, [~bbeaudreault] and [~zhangduo] 

I think the core issue lies in how \{{MetricsTableRequests}} register via \{{createRegistryForTableRequests}} by incrementing reference counts in \{{MetricRegistries.global()}}. When all regions shut down, the \{{removeRegistry}} method is invoked to remove \{{metricRegistryInfo}} from \{{MetricRegistries.global()}}. This causes a new \{{MetricRegistry}} to be initialized when the region comes back online. This \{{MetricRegistry}} is not registered in \{{DefaultMetricsSystem.instance()}}, as only the \{{MetricsSourceAdapter}} containing the old \{{MetricRegistry}} was previously registered in \{{GlobalMetricRegistriesAdapter}}. The invalid registry is not promptly removed because the \{{doRun}} method performs a 10-second polling check.

{code:language=bash|borderStyle=solid|theme=RDark|linenumbers=true|collapse=false}
1. Region closed
   └── MetricsTableRequests.removeRegistry()
       └── MetricRegistries.global().remove(info)
           └── RefCountingMap.remove() → refCount--
               └── When refCount == 0, MetricRegistry is removed from the map

2. Region reopens
   └── MetricsTableRequests.init()
       └── MetricRegistries.global().create(info)
           └── RefCountingMap.put() → refCount=1, creates new MetricRegistry instance

3. GlobalMetricRegistriesAdapter.doRun() (10 seconds later)
   └── Detects registry instance for info exists, takes no action
   └── Fails to update registeredSources and DefaultMetricsSystem, JMX still points to old registry (no data updated)
{code}

 

> TableRequests metrics broken after altering table
> -------------------------------------------------
>
>                 Key: HBASE-28123
>                 URL: https://issues.apache.org/jira/browse/HBASE-28123
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Bryan Beaudreault
>            Priority: Major
>              Labels: pull-request-available
>
> Alerting a table reopens all regions. I noticed after altering one of my tables that afterwards the TableRequests metrics no longer update. They seem stuck at whatever they were prior. I'm thinking this has something to do with the refactor done in https://issues.apache.org/jira/browse/HBASE-27681, which attempts to deregister metrics when a table is removed from the server. I think when it gets re-registered the new metrics aren't linked to whatever the jmx adapter is looking at.
> [~tianhang] FYI



--
This message was sent by Atlassian Jira
(v8.20.10#820010)