You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Levas <rl...@hortonworks.com> on 2015/02/26 13:02:16 UTC
Review Request 31483: Root user has spnego (HTTP) kerberos ticket set
after Kerberos is enabled, root should have no ticket.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
Bugs: AMBARI-9785
https://issues.apache.org/jira/browse/AMBARI-9785
Repository: ambari
Description
-------
After enabling Kerberos, the root user has the spnego user set for it
```
[root@c6501 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
Valid starting Expires Service principal
02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 02/18/15 22:14:51
```
It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
Diff: https://reviews.apache.org/r/31483/diff/
Testing
-------
Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
#Jenkins Test Results
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:12 h
[INFO] Finished at: 2015-02-26T06:35:45+00:00
[INFO] Final Memory: 44M/457M
[INFO] ------------------------------------------------------------------------
Thanks,
Robert Levas
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 8:02 a.m., Emil Anca wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py, lines 33-40
> > <https://reviews.apache.org/r/31483/diff/1/?file=878321#file878321line33>
> >
> > For readability purposes, this could be refactored to
> > [for]
> > if x: # x neither None nor empty
> > path = os.path.join(x,"klist")
> > if os.path.isfile(path):
> > klist_path = path
> > break
> > [/for]
>
> Robert Levas wrote:
> As much as I agrree with you and dislike the current implementation - it was copied and pasted from `get_kinit_path` and thus proven to not be buggy. So I would prefer not to change it until we make a more global fix as mentioned in comments on Andrew's review.
Fixing in next patch.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74261
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 8:02 a.m., Emil Anca wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py, lines 33-40
> > <https://reviews.apache.org/r/31483/diff/1/?file=878321#file878321line33>
> >
> > For readability purposes, this could be refactored to
> > [for]
> > if x: # x neither None nor empty
> > path = os.path.join(x,"klist")
> > if os.path.isfile(path):
> > klist_path = path
> > break
> > [/for]
As much as I agrree with you and dislike the current implementation - it was copied and pasted from `get_kinit_path` and thus proven to not be buggy. So I would prefer not to change it until we make a more global fix as mentioned in comments on Andrew's review.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74261
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Emil Anca <ea...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74261
-----------------------------------------------------------
Ship it!
Looks good.
ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py
<https://reviews.apache.org/r/31483/#comment120798>
For readability purposes, this could be refactored to
[for]
if x: # x neither None nor empty
path = os.path.join(x,"klist")
if os.path.isfile(path):
klist_path = path
break
[/for]
- Emil Anca
On Feb. 26, 2015, 12:02 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 12:02 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74565
-----------------------------------------------------------
Ship it!
ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py
<https://reviews.apache.org/r/31483/#comment121170>
I'm not sure what happens if self.config.get('agent', 'tmp_dir') is not defined; Maybe it always has a default from AmbariConfig. Might be safer to assign it outside of the function call and check for None. If it's None, then use tempfile.gettempdir()
- Jonathan Hurley
On Feb. 27, 2015, 1:26 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 27, 2015, 1:26 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py a53201e
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 5052ca0
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 6bcd7cd
> ambari-common/src/main/python/resource_management/libraries/functions/find_path.py PRE-CREATION
> ambari-common/src/main/python/resource_management/libraries/functions/get_kdestroy_path.py f6d7f78
> ambari-common/src/main/python/resource_management/libraries/functions/get_kinit_path.py 98434a2
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 9773ec3
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py 23a3482
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/package/scripts/params.py f368703
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py 9fbce1d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/params.py 90235e9
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/scripts/params.py f13fbaf
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/PIG/package/scripts/params.py b334f3b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/package/scripts/params.py 187ddc0
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/package/scripts/params.py b668f67
> ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/YARN/package/scripts/params.py 0fdaf18
> ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/FALCON/package/scripts/params.py 056380b
> ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/YARN/package/scripts/params.py 0fdaf18
> ambari-server/src/main/resources/stacks/HDP/2.2.GlusterFS/services/KNOX/package/scripts/params.py b11eddb
> ambari-server/src/main/resources/stacks/HDP/2.2.GlusterFS/services/SLIDER/package/scripts/params.py 121bbd4
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/YARN/package/scripts/params.py a548f98
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HBASE/package/scripts/params.py 8c4f5e5
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HDFS/package/scripts/params.py ff61dd7
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HIVE/package/scripts/params.py a9a67f7
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/NAGIOS/package/scripts/params.py 5a0ffbb
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/OOZIE/package/scripts/params.py 7659529
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/PIG/package/scripts/params.py 60a9806
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/YARN/package/scripts/params.py 6e7f170
> ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/ZOOKEEPER/package/scripts/params.py 9fa99b5
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 321a9c1
> ambari-server/src/test/resources/TestAmbaryServer.samples/dummy_common_services/HIVE/0.11.0.2.0.5.0/package/scripts/params.py 0cf89be
> ambari-server/src/test/resources/TestAmbaryServer.samples/dummy_stack/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HIVE/package/scripts/params.py 648913b
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/
-----------------------------------------------------------
(Updated Feb. 27, 2015, 1:26 p.m.)
Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
Changes
-------
Addresses reviewer concerns and applied changes due to merging conflicts
Bugs: AMBARI-9785
https://issues.apache.org/jira/browse/AMBARI-9785
Repository: ambari
Description
-------
After enabling Kerberos, the root user has the spnego user set for it
```
[root@c6501 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
Valid starting Expires Service principal
02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 02/18/15 22:14:51
```
It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/AlertSchedulerHandler.py a53201e
ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 5052ca0
ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 6bcd7cd
ambari-common/src/main/python/resource_management/libraries/functions/find_path.py PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/functions/get_kdestroy_path.py f6d7f78
ambari-common/src/main/python/resource_management/libraries/functions/get_kinit_path.py 98434a2
ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 9773ec3
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py 23a3482
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/package/scripts/params.py f368703
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py 9fbce1d
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/params.py 90235e9
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/scripts/params.py f13fbaf
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/PIG/package/scripts/params.py b334f3b
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/package/scripts/params.py 187ddc0
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/package/scripts/params.py b668f67
ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/YARN/package/scripts/params.py 0fdaf18
ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/FALCON/package/scripts/params.py 056380b
ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/YARN/package/scripts/params.py 0fdaf18
ambari-server/src/main/resources/stacks/HDP/2.2.GlusterFS/services/KNOX/package/scripts/params.py b11eddb
ambari-server/src/main/resources/stacks/HDP/2.2.GlusterFS/services/SLIDER/package/scripts/params.py 121bbd4
ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/YARN/package/scripts/params.py a548f98
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HBASE/package/scripts/params.py 8c4f5e5
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HDFS/package/scripts/params.py ff61dd7
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/HIVE/package/scripts/params.py a9a67f7
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/NAGIOS/package/scripts/params.py 5a0ffbb
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/OOZIE/package/scripts/params.py 7659529
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/PIG/package/scripts/params.py 60a9806
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/YARN/package/scripts/params.py 6e7f170
ambari-server/src/main/resources/stacks/PHD/3.0.0.0/services/ZOOKEEPER/package/scripts/params.py 9fa99b5
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 321a9c1
ambari-server/src/test/resources/TestAmbaryServer.samples/dummy_common_services/HIVE/0.11.0.2.0.5.0/package/scripts/params.py 0cf89be
ambari-server/src/test/resources/TestAmbaryServer.samples/dummy_stack/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HIVE/package/scripts/params.py 648913b
Diff: https://reviews.apache.org/r/31483/diff/
Testing
-------
Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
#Jenkins Test Results
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:12 h
[INFO] Finished at: 2015-02-26T06:35:45+00:00
[INFO] Final Memory: 44M/457M
[INFO] ------------------------------------------------------------------------
Thanks,
Robert Levas
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 7:09 a.m., Andrew Onischuk wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py, line 26
> > <https://reviews.apache.org/r/31483/diff/1/?file=878321#file878321line26>
> >
> > Can you create more generic function.
> > Something like:
> >
> > get_binary_path(pathes_list, binary_name):
> >
> > and use it as:
> > get_binary_path(..., "klist")
> > get_binary_path(..., "kinit")
> >
> > I know we already use the same approach for kinit, and maybe in future need some more like this.
>
> Robert Levas wrote:
> I was thinking about this as well and teetered back and forth on doing that. I decided to do the copy to try not to introduce a new bug. Also I know that we will be reworking the get_kinit_path and get_klist_path functions when we support alternate locations of kinit and klist. So I assumed we would do a similar fix then.
Between this issue and Emil's comment, I decided to bite the bullet and add. The fix will be in the next patch.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74255
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Feb. 26, 2015, 12:09 p.m., Andrew Onischuk wrote:
> > ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py, line 58
> > <https://reviews.apache.org/r/31483/diff/1/?file=878327#file878327line58>
> >
> > Tests will fail if klist is present on system. In tests we also have to mock call function you used, to return always 0 or 1.
just a small changes better I think better to do. Otherwise patch looks good.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74255
-----------------------------------------------------------
On Feb. 26, 2015, 12:02 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 12:02 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 7:09 a.m., Andrew Onischuk wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py, line 26
> > <https://reviews.apache.org/r/31483/diff/1/?file=878321#file878321line26>
> >
> > Can you create more generic function.
> > Something like:
> >
> > get_binary_path(pathes_list, binary_name):
> >
> > and use it as:
> > get_binary_path(..., "klist")
> > get_binary_path(..., "kinit")
> >
> > I know we already use the same approach for kinit, and maybe in future need some more like this.
I was thinking about this as well and teetered back and forth on doing that. I decided to do the copy to try not to introduce a new bug. Also I know that we will be reworking the get_kinit_path and get_klist_path functions when we support alternate locations of kinit and klist. So I assumed we would do a similar fix then.
> On Feb. 26, 2015, 7:09 a.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py, line 112
> > <https://reviews.apache.org/r/31483/diff/1/?file=878322#file878322line112>
> >
> > Can we evaluate klist path in params, to be able to do more usages in future is it's necessary
As mentioned in the previous comment, this will be addressed when we take on alterante paths for kinit and klist. I don't think such a change is appropriate now, espically with the current time constraints.
> On Feb. 26, 2015, 7:09 a.m., Andrew Onischuk wrote:
> > ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py, line 58
> > <https://reviews.apache.org/r/31483/diff/1/?file=878327#file878327line58>
> >
> > Tests will fail if klist is present on system. In tests we also have to mock call function you used, to return always 0 or 1.
>
> Andrew Onischuk wrote:
> just a small changes better I think better to do. Otherwise patch looks good.
I'm looking into this one... Python mocks are not my forte. Thanks for the review.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74255
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 7:09 a.m., Andrew Onischuk wrote:
> > ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py, line 58
> > <https://reviews.apache.org/r/31483/diff/1/?file=878327#file878327line58>
> >
> > Tests will fail if klist is present on system. In tests we also have to mock call function you used, to return always 0 or 1.
>
> Andrew Onischuk wrote:
> just a small changes better I think better to do. Otherwise patch looks good.
>
> Robert Levas wrote:
> I'm looking into this one... Python mocks are not my forte. Thanks for the review.
Unless I missunderstand you (you want me to mock up the `shell.as_user` function)... but I dont think any real calls happen and the `shell.as_user` call simply generates a string that is getting checked in the `assertResourceCalled` function. Therefore the existance of `klist` is illrelevant.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74255
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74255
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py
<https://reviews.apache.org/r/31483/#comment120786>
Can you create more generic function.
Something like:
get_binary_path(pathes_list, binary_name):
and use it as:
get_binary_path(..., "klist")
get_binary_path(..., "kinit")
I know we already use the same approach for kinit, and maybe in future need some more like this.
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py
<https://reviews.apache.org/r/31483/#comment120787>
Can we evaluate klist path in params, to be able to do more usages in future is it's necessary
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py
<https://reviews.apache.org/r/31483/#comment120790>
Tests will fail if klist is present on system. In tests we also have to mock call function you used, to return always 0 or 1.
- Andrew Onischuk
On Feb. 26, 2015, 12:02 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 12:02 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Feb. 26, 2015, 12:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py, line 109
> > <https://reviews.apache.org/r/31483/diff/1/?file=878322#file878322line109>
> >
> > Same as above; hard coding the /tmp path.
>
> Robert Levas wrote:
> Do you know how to get the `tmp_path` value from the `Environment` from this point in the code? I am not familair enough with this to figure it out. Or maybe `tmp_path` is in the `parameters` dictionary?
The agent has it's own tmp location configuration in /etc/ambari-agent/conf/ambari-agent.ini
```
[agent]
tmp_dir=/var/lib/ambari-agent/data/tmp
```
You can use this; it's accessed from AmbariConfig.py
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74285
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 12:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py, line 109
> > <https://reviews.apache.org/r/31483/diff/1/?file=878322#file878322line109>
> >
> > Same as above; hard coding the /tmp path.
Do you know how to get the `tmp_path` value from the `Environment` from this point in the code? I am not familair enough with this to figure it out. Or maybe `tmp_path` is in the `parameters` dictionary?
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74285
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Jeff Sposetti <je...@hortonworks.com>.
> On Feb. 26, 2015, 5:14 p.m., Jonathan Hurley wrote:
> > ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py, line 137
> > <https://reviews.apache.org/r/31483/diff/1/?file=878319#file878319line137>
> >
> > Hard coding /tmp ?
Believe {params.tmp_dir} was made available as part of https://issues.apache.org/jira/browse/AMBARI-6408
> On Feb. 26, 2015, 5:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py, line 109
> > <https://reviews.apache.org/r/31483/diff/1/?file=878322#file878322line109>
> >
> > Same as above; hard coding the /tmp path.
>
> Robert Levas wrote:
> Do you know how to get the `tmp_path` value from the `Environment` from this point in the code? I am not familair enough with this to figure it out. Or maybe `tmp_path` is in the `parameters` dictionary?
>
> Jonathan Hurley wrote:
> The agent has it's own tmp location configuration in /etc/ambari-agent/conf/ambari-agent.ini
>
> ```
> [agent]
> tmp_dir=/var/lib/ambari-agent/data/tmp
> ```
>
> You can use this; it's accessed from AmbariConfig.py
>
> Robert Levas wrote:
> Thanks... I think I found where the config is being passed around. I will have a new patch up shortly. Testing now.
Believe {params.tmp_dir} was made available as part of https://issues.apache.org/jira/browse/AMBARI-6408
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74285
-----------------------------------------------------------
On Feb. 26, 2015, 12:02 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 12:02 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 26, 2015, 12:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py, line 109
> > <https://reviews.apache.org/r/31483/diff/1/?file=878322#file878322line109>
> >
> > Same as above; hard coding the /tmp path.
>
> Robert Levas wrote:
> Do you know how to get the `tmp_path` value from the `Environment` from this point in the code? I am not familair enough with this to figure it out. Or maybe `tmp_path` is in the `parameters` dictionary?
>
> Jonathan Hurley wrote:
> The agent has it's own tmp location configuration in /etc/ambari-agent/conf/ambari-agent.ini
>
> ```
> [agent]
> tmp_dir=/var/lib/ambari-agent/data/tmp
> ```
>
> You can use this; it's accessed from AmbariConfig.py
Thanks... I think I found where the config is being passed around. I will have a new patch up shortly. Testing now.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74285
-----------------------------------------------------------
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31483: Root user has spnego (HTTP) kerberos ticket
set after Kerberos is enabled, root should have no ticket.
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31483/#review74285
-----------------------------------------------------------
Ship it!
ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py
<https://reviews.apache.org/r/31483/#comment120829>
Hard coding /tmp ?
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py
<https://reviews.apache.org/r/31483/#comment120831>
Same as above; hard coding the /tmp path.
- Jonathan Hurley
On Feb. 26, 2015, 7:02 a.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31483/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 7:02 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Emil Anca, Jonathan Hurley, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9785
> https://issues.apache.org/jira/browse/AMBARI-9785
>
>
> Repository: ambari
>
>
> Description
> -------
>
> After enabling Kerberos, the root user has the spnego user set for it
>
> ```
> [root@c6501 ~]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: HTTP/c6501.ambari.apache.org@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 02/18/15 22:14:51 02/19/15 22:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 02/18/15 22:14:51
> ```
>
> It appears that the issue is related to the agent-side scheduler and/or some job that is scheduled to run periodically. Apparently some job is kinit-ing with the SPNEGO identity as the running user (root in this case) without changing the ticket cache. Thus whenever the job runs the root user's ticket cache gets changed to contain the SPNEGO identity's ticket.
>
> While investigating and solving the issue it was found that other credentials were added to this cache, overwriting what was there, during backround processing, as well.
>
> Most of the issues were releated to _alert_ checking on web-based UI endpoints while configuring the environment for curl to use Kerberos authentication. Another place (in Oozie) was a failure to run a command as the `oozie` local user.
>
> Solving this includes using an alternate credential cache when kinit-ing. While at it, the cached is checked to see if the tickets are expired (or even there) before kinit-ing.
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/alerts/web_alert.py 8ee6606
> ambari-common/src/main/python/resource_management/libraries/functions/__init__.py 44d235c
> ambari-common/src/main/python/resource_management/libraries/functions/get_klist_path.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py 970ddde
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/alerts/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py 092149d
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/package/files/alert_check_oozie_server.py a5a066b
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/WEBHCAT/package/files/alert_webhcat_server.py 970ddde
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 45e9dc4
>
> Diff: https://reviews.apache.org/r/31483/diff/
>
>
> Testing
> -------
>
> Manually tested all services in test cluster to see which might have this issue. Found only OOZIE and HIVE issues and tests showed they are fixed and working as they should.
>
> #Jenkins Test Results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:12 h
> [INFO] Finished at: 2015-02-26T06:35:45+00:00
> [INFO] Final Memory: 44M/457M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Robert Levas
>
>