You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Sandor Magyari (JIRA)" <ji...@apache.org> on 2015/12/17 00:30:47 UTC
[jira] [Created] (AMBARI-14409) Blueprints: Kerberos deployments
fail intermittently due to invalid keytabs
Sandor Magyari created AMBARI-14409:
---------------------------------------
Summary: Blueprints: Kerberos deployments fail intermittently due to invalid keytabs
Key: AMBARI-14409
URL: https://issues.apache.org/jira/browse/AMBARI-14409
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.0.0
Reporter: Sandor Magyari
Assignee: Sandor Magyari
Priority: Critical
Fix For: 2.2.1
The basic problem appears to be a concurrency problem, in which keytabs are sometimes generated incorrectly. This results in certain keytabs across the cluster having different created/modified times in the keytabs, which can cause Kerberos Preauthentication failures, such as the following:
{code}
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-clusterone@EXAMPLE.COM;' returned 1. kinit: Password incorrect while getting initial credentials
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py", line 179, in <module>
ZookeeperServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 217, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py", line 55, in start
zookeeper_service(action='start', upgrade_type=upgrade_type)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py", line 54, in zookeeper_service
user=params.smokeuser
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
{code}
The basic underlying problem appears to be invalid keytabs. Looking to the keytabs on the failing node:
{code}
[root@rwntest-2 keytabs]# klist -kt smokeuser.headless.keytab
Keytab name: FILE:smokeuser.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
1 12/04/15 20:25:04 ambari-qa-clusterone@EXAMPLE.COM
{code}
whereas other nodes in the cluster report the following value for the same keytab:
{code}
[root@rwntest-4 keytabs]# klist -kt smokeuser.headless.keytab
Keytab name: FILE:smokeuser.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
4 12/04/15 20:25:32 ambari-qa-clusterone@EXAMPLE.COM
{code}
It looks like these keytabs should have the same timestamps, and this is causing the Kerberos failures at cluster start up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)