You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ambari.apache.org by Fabien Carrion <fa...@gmail.com> on 2019/01/10 05:57:04 UTC

Ambari Hbase Failed Regions

Hello,

I need to create an ambari' alert if I have a failed region.

I didnt see this metric in the widget creator so I was thinking to
create an ambari metrics, and then create an ambari alert on it.

I would like to know if there is a way to access to this hbase metric
through ganglia or if I need to create a whole new python script to
download the metric.

As I am new to ambari, I am open to any suggestion.

Thanks for the help

-- 
Fabien Carrion

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/

Re: Ambari Hbase Failed Regions

Posted by Fabien Carrion <fa...@gmail.com>.

Thanks for the help. Failed regions is not exposed through JMX, so I
just did a small script to download it from hbase master status webpage:

failedRegionsCount=$(curl -s http://127.0.0.1:16010/master-status |
hxnormalize -x | hxselect -i '#tab_userTables' | w3m -dump -cols 2000 -T
'text/html' | awk '{print $3" "$4" "$5" "$6" "$7}' | grep -v Details |
grep -v Regions | awk '{ sum+=$3} END {print sum}')
timestamp=$(date +%s)

I thought a quick curl would be easier to setup than a python script. In
case that's the root cause of my problem, I'll switch.

Then I added the metric in
/var/lib/ambari-server/resources/stacks/HDP/2.3/services/HBASE/metrics.json


  "HBASE_MASTER": {
    "Component": [
      {
        "type": "ganglia",
        "metrics": {
          "default": {
            "metrics/clic2buy": {
              "metric": "FailedRegionsCount",
              "pointInTime": true,
              "temporal": true
            },


It appears in
http://localhost:8080/api/v1/stacks/HDP/versions/2.6/services/HBASE/artifacts/metrics_descriptor

I can also create a widget based on that metric.

But I cant send data to it. I used the REST api of the metric collector:

curl -vvv -H "Content-Type: application/json" -X POST -d '{"metrics":
[{"metricname": "FailedRegionsCount", "appid": "hbase", "hostname":
"sandbox.test.com", "timestamp": '"$timestamp"', "starttime":
'"$timestamp"', "metrics": {"'"$timestamp"'":
'"$failedRegionsCount"'}}]}' "http://127.0.0.1:6188/ws/v1/timeline/metrics"

=> HTTP status 200

where sandbox.test.com is the domain of my hbase master.

A HTTP GET dont show data either:

curl -vvv -H "Content-Type: application/json" -X GET
"http://127.0.0.1:6188/ws/v1/timeline/metrics?metricNames=FailedRegionsCount&appId=hbase"

=> HTTP status 200
{"metrics":[]}

But I can load data from metrics load_one with the same GET request for
example.

I would say I forgot to initialized something so the metric declared in
ambari server would be propaged to ambari metric collector, but I am not
sure what it is. In case that's python is required, well let me know.

Thanks for the help

On 1/11/19 7:53 AM, Jonathan Hurley wrote:
> Ambari alerts work natively off of JMX metrics, so if HBase exposes a metric through JMX then you can consume it. We used to do something similar for "dead" RegionServers, but it was determined that a "dead" RegionServer was part of the normal lifecycle and shouldn't have an alert: 
> 
> https://github.com/apache/ambari/commit/d1e59133f7f867a16d6cb4c7e5cc8508893a29fe#diff-c6c19e2641a79e05745b9320dfb7a6e1
> 
> However, if the metric you want isn't exposed by HBase, then you'd have to write a Python script to do the work for you.
> 
> On 1/10/19, 1:04 AM, "Fabien Carrion" <fa...@gmail.com> wrote:
> 
>     Hello,
>     
>     I need to create an ambari' alert if I have a failed region.
>     
>     I didnt see this metric in the widget creator so I was thinking to
>     create an ambari metrics, and then create an ambari alert on it.
>     
>     I would like to know if there is a way to access to this hbase metric
>     through ganglia or if I need to create a whole new python script to
>     download the metric.
>     
>     As I am new to ambari, I am open to any suggestion.
>     
>     Thanks for the help
>     
>     -- 
>     Fabien Carrion
>     
>     ()  Campagne du ruban ASCII -- Contre les mails en html
>     /\  contre les pieces-jointes Microsoft
>     Web: http://fabien.carrion.free.fr/
>     
>     
> 

-- 
Fabien Carrion

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/



-- 
Fabien Carrion

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/

Re: Ambari Hbase Failed Regions

Posted by Jonathan Hurley <jh...@hortonworks.com>.

Ambari alerts work natively off of JMX metrics, so if HBase exposes a metric through JMX then you can consume it. We used to do something similar for "dead" RegionServers, but it was determined that a "dead" RegionServer was part of the normal lifecycle and shouldn't have an alert: 

https://github.com/apache/ambari/commit/d1e59133f7f867a16d6cb4c7e5cc8508893a29fe#diff-c6c19e2641a79e05745b9320dfb7a6e1

However, if the metric you want isn't exposed by HBase, then you'd have to write a Python script to do the work for you.

On 1/10/19, 1:04 AM, "Fabien Carrion" <fa...@gmail.com> wrote:

    Hello,
    
    I need to create an ambari' alert if I have a failed region.
    
    I didnt see this metric in the widget creator so I was thinking to
    create an ambari metrics, and then create an ambari alert on it.
    
    I would like to know if there is a way to access to this hbase metric
    through ganglia or if I need to create a whole new python script to
    download the metric.
    
    As I am new to ambari, I am open to any suggestion.
    
    Thanks for the help
    
    -- 
    Fabien Carrion
    
    ()  Campagne du ruban ASCII -- Contre les mails en html
    /\  contre les pieces-jointes Microsoft
    Web: http://fabien.carrion.free.fr/