You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "Istvan Toth (Jira)" <ji...@apache.org> on 2022/04/28 05:37:00 UTC
[jira] [Comment Edited] (PHOENIX-6699) Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
[ https://issues.apache.org/jira/browse/PHOENIX-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529205#comment-17529205 ]
Istvan Toth edited comment on PHOENIX-6699 at 4/28/22 5:36 AM:
---------------------------------------------------------------
To elaborate:
We saw this (filtered) log on a Region server:
{noformat}
2022-04-26 15:09:18,233 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system started
2022-04-26 15:09:22,619 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping HBase metrics system...
2022-04-26 15:09:22,624 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system stopped.
2022-04-26 15:09:23,136 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system started
2022-04-26 15:09:27,863 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping HBase metrics system...
2022-04-26 15:09:27,880 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system stopped.
2022-04-26 15:09:27,891 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started
2022-04-26 15:09:28,380 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system already started!
2022-04-26 15:09:32,786 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping Phoenix metrics system...
2022-04-26 15:09:32,840 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system stopped.
2022-04-26 15:09:33,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started
2022-04-26 15:09:37,904 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping Phoenix metrics system...
2022-04-26 15:09:37,909 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system stopped.
2022-04-26 15:09:38,411 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started{noformat}
The HBase metrics messages are caused by JmxCacheBuster.
then we see that 10 ms after JMXCacheBuster has stopped the DefaultMetricsSystem, the "Phoenix" metrics system gets registered.
This happens, because while DefaultMetricsSystem.init() will not overwrite the instance while it's running, init() WILL overwrite the instance when it's stopped (by JMXCachecBuster).
The _Phoenix metrics system already started! **_ line is printed when JMXCacheBuster re-starts the metrics system, however, by this time it has been overwritten by Phoenix.
was (Author: stoty):
To elaborate:
We saw this (filtered) log on a Region server:
{noformat}
2022-04-26 15:09:18,233 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system started
2022-04-26 15:09:22,619 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping HBase metrics system...
2022-04-26 15:09:22,624 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system stopped.
2022-04-26 15:09:23,136 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system started
2022-04-26 15:09:27,863 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping HBase metrics system...
2022-04-26 15:09:27,880 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: HBase metrics system stopped.
2022-04-26 15:09:27,891 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started
2022-04-26 15:09:28,380 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system already started!
2022-04-26 15:09:32,786 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping Phoenix metrics system...
2022-04-26 15:09:32,840 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system stopped.
2022-04-26 15:09:33,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started
2022-04-26 15:09:37,904 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping Phoenix metrics system...
2022-04-26 15:09:37,909 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system stopped.
2022-04-26 15:09:38,411 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system started{noformat}
The HBase metrics messages are caused by JmxCacheBuster.
then we see that 10 ms after JMXCacheBuster has stopped the DefaultMetricsSystem, the "Phoenix" metrics system gets registered.
This happens, because while DefaultMetricsSystem.init() will not overwrite the instance while it's running, init() WILL overwrite the instance when it's stopped (by JMXCachecBuster).
> Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
> -----------------------------------------------------------------
>
> Key: PHOENIX-6699
> URL: https://issues.apache.org/jira/browse/PHOENIX-6699
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> There is a race condition in the Hbase and Phoenix metrics handling.
> Hbase metrics is supposed to run in the daemon processes, while Phoenix metrics is supposed to run on the phoenix client side.
> On an RS both HBase and Phoenix tries to register its metrics implementation into DefaultMetricsSystem.
>
> On the happy path, when Phoenix tries to register its metrics on the RS, it won't have any effect, because HBase has already registered an active Metrics implementation.
> However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five seconds, and if Phoenix tries to initialize its metrics in this period, then it will succeed in initializing its Metrics object, and will overwrite theDefaultMetricsSystem object initialized by HBase.
> This disables the HBase metrics system for the RS process.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)