[jira] [Created] (AMBARI-25101) HBase metrics not available

["oktay tuncay (JIRA)" <ji...@apache.org>]

oktay tuncay created AMBARI-25101:
-------------------------------------

             Summary: HBase metrics not available
                 Key: AMBARI-25101
                 URL: https://issues.apache.org/jira/browse/AMBARI-25101
             Project: Ambari
          Issue Type: Bug
          Components: ambari-metrics
    Affects Versions: 2.7.1
            Reporter: oktay tuncay
         Attachments: AbstractTimelineMetricsSink-diff.png

TL;DR: I believe this is a bug introduced by the change of the class responsible for fetching the available AMS Collectors. This bug occurs when using more than one AMS Collector, details below.

-------------------------------------------------------------------------
This issue affects all metrics that use org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.

After increasing the verbosity of loggin to DEBUG for the class org.apache.hadoop.metrics2,*, it is possible to see that the metrics are not being sent due a malformed URI in the connection string:

The malformed URI is composed by both AMS Collectors adresses separated by comma, so I removed one of them as a test and restarted the services. After this no more errors were shown in the logs and the metrics were sent to the AMS DB (they weren't present before). Also, the dashboards started to show some graphs.

Since this configuration was working in the previous versions (HDP 2.6 + Ambari 2.5) and, according to documentation, is still supported[1] with the new versions (HDP 3.0 + Ambari 2.7) I checked the code of HadoopTimelineMetricsSink[2] to verify what could be causing this. By analyzing the code and correlating with the error messages in the log, we can see that the connection process is handled by another class: AbstractTimelineMetricsSink.

The AbstractTimelineMetricsSink had modifications from Ambari 2.5[3] to Ambari 2.7[4]. The following snippet highlights the main differences in the connection handling methods used by HadoopTimelineMetricsSink.

Based on that, I believe this is a bug introduced by the change of AbstractTimelinetMetricsSink class. This bug occurs when using more than one AMS Collector.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)