You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Robert Levas <rl...@hortonworks.com> on 2017/03/05 21:38:19 UTC
Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
Bugs: AMBARI-20309
https://issues.apache.org/jira/browse/AMBARI-20309
Repository: ambari
Description
-------
HBase Master CPU Utilization Alert is in unknown state due to kinit error:
```
Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
```
This issue is also seen in /var/log/krb5kdc.log:
```
Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
```
#Cause
It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
```
{
"name": "hbase_master_cpu",
"label": "HBase Master CPU Utilization",
"description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
"interval": 5,
"scope": "ANY",
"enabled": true,
"source": {
"type": "METRIC",
"uri": {
"http": "{{hbase-site/hbase.master.info.port}}",
"default_port": 60010,
"connection_timeout": 5.0,
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
},
"reporting": {
"ok": {
"text": "{1} CPU, load {0:.1%}"
},
"warning": {
"text": "{1} CPU, load {0:.1%}",
"value": 200
},
"critical": {
"text": "{1} CPU, load {0:.1%}",
"value": 250
},
"units" : "%",
"type": "PERCENT"
},
"jmx": {
"property_list": [
"java.lang:type=OperatingSystem/SystemCpuLoad",
"java.lang:type=OperatingSystem/AvailableProcessors"
],
"value": "{0} * 100"
}
}
}
```
Notice:
```
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
#Solution
Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
```
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
Diff: https://reviews.apache.org/r/57324/diff/1/
Testing
-------
Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
# Local test results: PASSED
# Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review168010
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On March 6, 2017, 11:25 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 6, 2017, 11:25 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/2/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/
-----------------------------------------------------------
(Updated March 6, 2017, 12:26 p.m.)
Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
Bugs: AMBARI-20309
https://issues.apache.org/jira/browse/AMBARI-20309
Repository: ambari
Description
-------
HBase Master CPU Utilization Alert is in unknown state due to kinit error:
```
Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
```
This issue is also seen in /var/log/krb5kdc.log:
```
Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
```
#Cause
It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
```
{
"name": "hbase_master_cpu",
"label": "HBase Master CPU Utilization",
"description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
"interval": 5,
"scope": "ANY",
"enabled": true,
"source": {
"type": "METRIC",
"uri": {
"http": "{{hbase-site/hbase.master.info.port}}",
"default_port": 60010,
"connection_timeout": 5.0,
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
},
"reporting": {
"ok": {
"text": "{1} CPU, load {0:.1%}"
},
"warning": {
"text": "{1} CPU, load {0:.1%}",
"value": 200
},
"critical": {
"text": "{1} CPU, load {0:.1%}",
"value": 250
},
"units" : "%",
"type": "PERCENT"
},
"jmx": {
"property_list": [
"java.lang:type=OperatingSystem/SystemCpuLoad",
"java.lang:type=OperatingSystem/AvailableProcessors"
],
"value": "{0} * 100"
}
}
}
```
Notice:
```
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
#Solution
Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
```
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
Diff: https://reviews.apache.org/r/57324/diff/2/
Testing (updated)
-------
Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
# Local test results:
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:07.123s
[INFO] Finished at: Mon Mar 06 12:25:32 EST 2017
[INFO] Final Memory: 70M/596M
[INFO] ------------------------------------------------------------------------
```
# Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review168001
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On March 6, 2017, 4:25 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 6, 2017, 4:25 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/2/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/
-----------------------------------------------------------
(Updated March 6, 2017, 11:25 a.m.)
Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
Changes
-------
Changed code that fixes the existing JSON string to manipiulate JSON data rather than replace string data.
Bugs: AMBARI-20309
https://issues.apache.org/jira/browse/AMBARI-20309
Repository: ambari
Description
-------
HBase Master CPU Utilization Alert is in unknown state due to kinit error:
```
Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
```
This issue is also seen in /var/log/krb5kdc.log:
```
Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
```
#Cause
It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
```
{
"name": "hbase_master_cpu",
"label": "HBase Master CPU Utilization",
"description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
"interval": 5,
"scope": "ANY",
"enabled": true,
"source": {
"type": "METRIC",
"uri": {
"http": "{{hbase-site/hbase.master.info.port}}",
"default_port": 60010,
"connection_timeout": 5.0,
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
},
"reporting": {
"ok": {
"text": "{1} CPU, load {0:.1%}"
},
"warning": {
"text": "{1} CPU, load {0:.1%}",
"value": 200
},
"critical": {
"text": "{1} CPU, load {0:.1%}",
"value": 250
},
"units" : "%",
"type": "PERCENT"
},
"jmx": {
"property_list": [
"java.lang:type=OperatingSystem/SystemCpuLoad",
"java.lang:type=OperatingSystem/AvailableProcessors"
],
"value": "{0} * 100"
}
}
}
```
Notice:
```
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
#Solution
Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
```
"kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
"kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
```
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
Diff: https://reviews.apache.org/r/57324/diff/2/
Changes: https://reviews.apache.org/r/57324/diff/1-2/
Testing
-------
Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
# Local test results: PASSED
# Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167961
-----------------------------------------------------------
Ship it!
Ship It!
- Sebastian Toader
On March 5, 2017, 10:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 10:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Laszlo Puskas <lp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167985
-----------------------------------------------------------
Ship it!
Ship It!
- Laszlo Puskas
On March 5, 2017, 9:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 9:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 6, 2017, 9:16 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> > Lines 220-230 (patched)
> > <https://reviews.apache.org/r/57324/diff/1/?file=1656448#file1656448line220>
> >
> > That's one way of doing it :)
> >
> > I prefer the more direct JSON approach, similar to:
> >
> > ```
> > String source = atlasMetadataServerWebUI.getSource();
> > JsonObject sourceJson = new JsonParser().parse(source).getAsJsonObject();
> >
> > JsonObject uriJson = sourceJson.get("uri").getAsJsonObject();
> > uriJson.remove("kerberos_keytab");
> > uriJson.remove("kerberos_principal");
> > uriJson.addProperty("kerberos_keytab", "{{cluster-env/smokeuser_keytab}}");
> > uriJson.addProperty("kerberos_principal", "{{cluster-env/smokeuser_principal_name}}");
> > ```
> >
> > As long as you don't see a way for this to corrupt the JSON, then it's fine.
>
> Robert Levas wrote:
> I though about this, but this way we only change the particilar issue at hand. I am not sure if a user would have changed/fixed this manually already and I didn't want to blindly undo it. However, it we think that it unlikely that the original value would have been changed, I am open to go the explicit JSON route.
Totally makes sense ... however, perhaps you could verify the JSON after saving it to ensure that it can be de-serialized from the DB? Just to be safe since we're doing a string replace on a structured syntax.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167976
-----------------------------------------------------------
On March 5, 2017, 4:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 4:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Robert Levas <rl...@hortonworks.com>.
> On March 6, 2017, 9:16 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> > Lines 220-230 (patched)
> > <https://reviews.apache.org/r/57324/diff/1/?file=1656448#file1656448line220>
> >
> > That's one way of doing it :)
> >
> > I prefer the more direct JSON approach, similar to:
> >
> > ```
> > String source = atlasMetadataServerWebUI.getSource();
> > JsonObject sourceJson = new JsonParser().parse(source).getAsJsonObject();
> >
> > JsonObject uriJson = sourceJson.get("uri").getAsJsonObject();
> > uriJson.remove("kerberos_keytab");
> > uriJson.remove("kerberos_principal");
> > uriJson.addProperty("kerberos_keytab", "{{cluster-env/smokeuser_keytab}}");
> > uriJson.addProperty("kerberos_principal", "{{cluster-env/smokeuser_principal_name}}");
> > ```
> >
> > As long as you don't see a way for this to corrupt the JSON, then it's fine.
I though about this, but this way we only change the particilar issue at hand. I am not sure if a user would have changed/fixed this manually already and I didn't want to blindly undo it. However, it we think that it unlikely that the original value would have been changed, I am open to go the explicit JSON route.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167976
-----------------------------------------------------------
On March 5, 2017, 4:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 4:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Robert Levas <rl...@hortonworks.com>.
> On March 6, 2017, 9:16 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> > Lines 220-230 (patched)
> > <https://reviews.apache.org/r/57324/diff/1/?file=1656448#file1656448line220>
> >
> > That's one way of doing it :)
> >
> > I prefer the more direct JSON approach, similar to:
> >
> > ```
> > String source = atlasMetadataServerWebUI.getSource();
> > JsonObject sourceJson = new JsonParser().parse(source).getAsJsonObject();
> >
> > JsonObject uriJson = sourceJson.get("uri").getAsJsonObject();
> > uriJson.remove("kerberos_keytab");
> > uriJson.remove("kerberos_principal");
> > uriJson.addProperty("kerberos_keytab", "{{cluster-env/smokeuser_keytab}}");
> > uriJson.addProperty("kerberos_principal", "{{cluster-env/smokeuser_principal_name}}");
> > ```
> >
> > As long as you don't see a way for this to corrupt the JSON, then it's fine.
>
> Robert Levas wrote:
> I though about this, but this way we only change the particilar issue at hand. I am not sure if a user would have changed/fixed this manually already and I didn't want to blindly undo it. However, it we think that it unlikely that the original value would have been changed, I am open to go the explicit JSON route.
>
> Jonathan Hurley wrote:
> Totally makes sense ... however, perhaps you could verify the JSON after saving it to ensure that it can be de-serialized from the DB? Just to be safe since we're doing a string replace on a structured syntax.
I went with your approach with some verification that Ambari only updates the JSON if the expected data exists to begin with. See rev 2.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167976
-----------------------------------------------------------
On March 5, 2017, 4:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 4:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57324: HBase Master CPU Utilization Alert is in
unknown state due to kinit error
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57324/#review167976
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
Lines 220-230 (patched)
<https://reviews.apache.org/r/57324/#comment239979>
That's one way of doing it :)
I prefer the more direct JSON approach, similar to:
```
String source = atlasMetadataServerWebUI.getSource();
JsonObject sourceJson = new JsonParser().parse(source).getAsJsonObject();
JsonObject uriJson = sourceJson.get("uri").getAsJsonObject();
uriJson.remove("kerberos_keytab");
uriJson.remove("kerberos_principal");
uriJson.addProperty("kerberos_keytab", "{{cluster-env/smokeuser_keytab}}");
uriJson.addProperty("kerberos_principal", "{{cluster-env/smokeuser_principal_name}}");
```
As long as you don't see a way for this to corrupt the JSON, then it's fine.
- Jonathan Hurley
On March 5, 2017, 4:38 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57324/
> -----------------------------------------------------------
>
> (Updated March 5, 2017, 4:38 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Bal�zs Bence S�ri, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, Sebastian Toader, and Sid Wagle.
>
>
> Bugs: AMBARI-20309
> https://issues.apache.org/jira/browse/AMBARI-20309
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase Master CPU Utilization Alert is in unknown state due to kinit error:
>
> ```
> Execution of '/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/metric_alert_ambari-qa_cc_56787c2122a8214ca9775f3433361f8b -kt HTTP/_HOST@EXAMPLE.COM /etc/security/keytabs/spnego.service.keytab > /dev/null' returned 1. kinit: Client not found in Kerberos database while getting initial credentials
> ```
>
> This issue is also seen in /var/log/krb5kdc.log:
>
> ```
> Mar 03 16:43:06 c6401.ambari.apache.org krb5kdc[4749](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.64.101: CLIENT_NOT_FOUND: /etc/security/keytabs/spnego.service.keytab@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Client not found in Kerberos database
> ```
>
> #Cause
> It appears that the HBASE alerts.json file (`common-services/HBASE/0.96.0.2.0/alerts.json`) has swapped values for the `kerberos_keytab` and `kerberos_principal` properties.
>
> ```
> {
> "name": "hbase_master_cpu",
> "label": "HBase Master CPU Utilization",
> "description": "This host-level alert is triggered if CPU utilization of the HBase Master exceeds certain warning and critical thresholds. It checks the HBase Master JMX Servlet for the SystemCPULoad property. The threshold values are in percent.",
> "interval": 5,
> "scope": "ANY",
> "enabled": true,
> "source": {
> "type": "METRIC",
> "uri": {
> "http": "{{hbase-site/hbase.master.info.port}}",
> "default_port": 60010,
> "connection_timeout": 5.0,
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> },
> "reporting": {
> "ok": {
> "text": "{1} CPU, load {0:.1%}"
> },
> "warning": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 200
> },
> "critical": {
> "text": "{1} CPU, load {0:.1%}",
> "value": 250
> },
> "units" : "%",
> "type": "PERCENT"
> },
> "jmx": {
> "property_list": [
> "java.lang:type=OperatingSystem/SystemCpuLoad",
> "java.lang:type=OperatingSystem/AvailableProcessors"
> ],
> "value": "{0} * 100"
> }
> }
> }
> ```
>
> Notice:
> ```
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
> #Solution
> Fix values for the `kerberos_keytab` and `kerberos_principal` properties in `common-services/HBASE/0.96.0.2.0/alerts.json`:
>
> ```
> "kerberos_principal": "{{hbase-site/hbase.security.authentication.spnego.kerberos.principal}}",
> "kerberos_keytab": "{{hbase-site/hbase.security.authentication.spnego.kerberos.keytab}}"
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java 2a684dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/alerts.json 1b3ae25
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java 7ee66ef
>
>
> Diff: https://reviews.apache.org/r/57324/diff/1/
>
>
> Testing
> -------
>
> Manually tested in new Ambari 2.5.0 cluster and upgrade scenario from Ambari 2.4.2 to Ambari 2.5.0
>
> # Local test results: PASSED
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>