You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Levas <rl...@hortonworks.com> on 2015/07/12 16:19:09 UTC

Review Request 36433: kinit of hdfs Kerberos identity fails when starting added service(s) after upgrade to Ambari 2.1.0

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36433/
-----------------------------------------------------------

Review request for Ambari, Jonathan Hurley, John Speidel, Mahadev Konar, Robert Nettleton, and Tom Beerbower.


Bugs: AMBARI-12356
    https://issues.apache.org/jira/browse/AMBARI-12356


Repository: ambari


Description
-------

STR:
1. Install old version of ambari (2.0.1)
2. Enable security
3. Do Ambari only upgrade to ambari2.1.0
4. Add some component - HiveServer2 or Ooozie server
5. Try to start added component

Actual result:
Start have been failed. 

```
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 182, in <module>
    HiveServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", 

line 216, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 83, in start
    self.configure(env) # FOR SECURITY
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 54, in configure
    hive(name='hiveserver2')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in 

thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 127, in hive
    mode=params.webhcat_hdfs_user_mode
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 

__init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 390, in 

action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 387, in 

action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 236, in 

action_delayed
    main_resource.kinit()
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 416, in kinit
    user=user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 

__init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", 

line 254, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in 

inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in 

checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in 

_call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in 

_call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 

/etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM' returned 1. kinit: Keytab 

contains no suitable keys for hdfs@EXAMPLE.COM while getting initial credentials
```

Expected results:
Can start all added components.

# Cause
The Kerberos Descriptor structure changed between Ambari 2.0 and Ambari 2.1.  This change moved the "hdfs" Kerberos identity descriptor from the _global_ scope to under the HDFS service. After upgrading from Ambari 2.0 to Ambari 2.1  an additional "hdfs" Kerberos identity descriptor was added with the new principal name pattern - ${hadoop-env/hdfs_user}-${cluster_name}@${realm}.  This occurred because the stored Kerberos Descriptor contained the _old_ structure, and when Ambari generated a composite Kerberos Descriptor made up of the Kerberos Descriptor compiled from the relevant stack definition with stored changes applied, that additional "hdfs" Kerberos identity descriptor was added.  Because if this, the Kerberos logic became _confused_ and overwrote the existing hdfs keytab file with one that contained the new principal name.

# Solution
While migrating Ambari 2.0 to Ambari 2.1, fix the stored Kerberos Descriptor structure to match the new version's structure.


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ArtifactResourceProvider.java 680f9b8 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ArtifactDAO.java 27346dd 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java 849a938 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java 3d4d701 
  ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ArtifactResourceProviderTest.java 789fb54 
  ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog210Test.java 8708047 
  ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_no_hdfs.json PRE-CREATION 
  ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_simple.json PRE-CREATION 

Diff: https://reviews.apache.org/r/36433/diff/


Testing
-------

Manually tested

#Local test results: PASSED

#Jenkins test results: PENDING


Thanks,

Robert Levas


Re: Review Request 36433: kinit of hdfs Kerberos identity fails when starting added service(s) after upgrade to Ambari 2.1.0

Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36433/#review91421
-----------------------------------------------------------

Ship it!



ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java (line 64)
<https://reviews.apache.org/r/36433/#comment144785>

    :)


- Jonathan Hurley


On July 12, 2015, 11:41 a.m., Robert Levas wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36433/
> -----------------------------------------------------------
> 
> (Updated July 12, 2015, 11:41 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, John Speidel, Mahadev Konar, Robert Nettleton, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-12356
>     https://issues.apache.org/jira/browse/AMBARI-12356
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> STR:
> 1. Install old version of ambari (2.0.1)
> 2. Enable security
> 3. Do Ambari only upgrade to ambari2.1.0
> 4. Add some component - HiveServer2 or Ooozie server
> 5. Try to start added component
> 
> Actual result:
> Start have been failed. 
> 
> ```
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 182, in <module>
>     HiveServer().execute()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", 
> 
> line 216, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 83, in start
>     self.configure(env) # FOR SECURITY
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 54, in configure
>     hive(name='hiveserver2')
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in 
> 
> thunk
>     return fn(*args, **kwargs)
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 127, in hive
>     mode=params.webhcat_hdfs_user_mode
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 
> 
> __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 390, in 
> 
> action_create_on_execute
>     self.action_delayed("create")
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 387, in 
> 
> action_delayed
>     self.get_hdfs_resource_executor().action_delayed(action_name, self)
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 236, in 
> 
> action_delayed
>     main_resource.kinit()
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 416, in kinit
>     user=user
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 
> 
> __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", 
> 
> line 254, in action_run
>     tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in 
> 
> inner
>     result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in 
> 
> checked_call
>     tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in 
> 
> _call_wrapper
>     result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in 
> 
> _call
>     raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> 
> /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM' returned 1. kinit: Keytab 
> 
> contains no suitable keys for hdfs@EXAMPLE.COM while getting initial credentials
> ```
> 
> Expected results:
> Can start all added components.
> 
> # Cause
> The Kerberos Descriptor structure changed between Ambari 2.0 and Ambari 2.1.  This change moved the "hdfs" Kerberos identity descriptor from the _global_ scope to under the HDFS service. After upgrading from Ambari 2.0 to Ambari 2.1  an additional "hdfs" Kerberos identity descriptor was added with the new principal name pattern - ${hadoop-env/hdfs_user}-${cluster_name}@${realm}.  This occurred because the stored Kerberos Descriptor contained the _old_ structure, and when Ambari generated a composite Kerberos Descriptor made up of the Kerberos Descriptor compiled from the relevant stack definition with stored changes applied, that additional "hdfs" Kerberos identity descriptor was added.  Because if this, the Kerberos logic became _confused_ and overwrote the existing hdfs keytab file with one that contained the new principal name.
> 
> # Solution
> While migrating Ambari 2.0 to Ambari 2.1, fix the stored Kerberos Descriptor structure to match the new version's structure.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ArtifactResourceProvider.java 680f9b8 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ArtifactDAO.java 27346dd 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java 849a938 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java 3d4d701 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ArtifactResourceProviderTest.java 789fb54 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog210Test.java 8708047 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_no_hdfs.json PRE-CREATION 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_simple.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/36433/diff/
> 
> 
> Testing
> -------
> 
> Manually tested
> 
> #Local test results: PASSED
> 
> #Jenkins test results: 
> 
> Running org.apache.ambari.server.controller.internal.ArtifactResourceProviderTest
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.23 sec
> 
> Running org.apache.ambari.server.upgrade.UpgradeCatalog210Test
> Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.795 sec
> 
> Tests run: 3117, Failures: 0, Errors: 0, Skipped: 28
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:20 h
> [INFO] Finished at: 2015-07-12T15:37:50+00:00
> [INFO] Final Memory: 47M/659M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Robert Levas
> 
>


Re: Review Request 36433: kinit of hdfs Kerberos identity fails when starting added service(s) after upgrade to Ambari 2.1.0

Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36433/
-----------------------------------------------------------

(Updated July 12, 2015, 11:41 a.m.)


Review request for Ambari, Jonathan Hurley, John Speidel, Mahadev Konar, Robert Nettleton, and Tom Beerbower.


Bugs: AMBARI-12356
    https://issues.apache.org/jira/browse/AMBARI-12356


Repository: ambari


Description
-------

STR:
1. Install old version of ambari (2.0.1)
2. Enable security
3. Do Ambari only upgrade to ambari2.1.0
4. Add some component - HiveServer2 or Ooozie server
5. Try to start added component

Actual result:
Start have been failed. 

```
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 182, in <module>
    HiveServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", 

line 216, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 83, in start
    self.configure(env) # FOR SECURITY
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 54, in configure
    hive(name='hiveserver2')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in 

thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-

services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 127, in hive
    mode=params.webhcat_hdfs_user_mode
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 

__init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 390, in 

action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 387, in 

action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 236, in 

action_delayed
    main_resource.kinit()
  File "/usr/lib/python2.6/site-

packages/resource_management/libraries/providers/hdfs_resource.py", line 416, in kinit
    user=user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 

__init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 

118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", 

line 254, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in 

inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in 

checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in 

_call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in 

_call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 

/etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM' returned 1. kinit: Keytab 

contains no suitable keys for hdfs@EXAMPLE.COM while getting initial credentials
```

Expected results:
Can start all added components.

# Cause
The Kerberos Descriptor structure changed between Ambari 2.0 and Ambari 2.1.  This change moved the "hdfs" Kerberos identity descriptor from the _global_ scope to under the HDFS service. After upgrading from Ambari 2.0 to Ambari 2.1  an additional "hdfs" Kerberos identity descriptor was added with the new principal name pattern - ${hadoop-env/hdfs_user}-${cluster_name}@${realm}.  This occurred because the stored Kerberos Descriptor contained the _old_ structure, and when Ambari generated a composite Kerberos Descriptor made up of the Kerberos Descriptor compiled from the relevant stack definition with stored changes applied, that additional "hdfs" Kerberos identity descriptor was added.  Because if this, the Kerberos logic became _confused_ and overwrote the existing hdfs keytab file with one that contained the new principal name.

# Solution
While migrating Ambari 2.0 to Ambari 2.1, fix the stored Kerberos Descriptor structure to match the new version's structure.


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ArtifactResourceProvider.java 680f9b8 
  ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ArtifactDAO.java 27346dd 
  ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java 849a938 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java 3d4d701 
  ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ArtifactResourceProviderTest.java 789fb54 
  ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog210Test.java 8708047 
  ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_no_hdfs.json PRE-CREATION 
  ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_simple.json PRE-CREATION 

Diff: https://reviews.apache.org/r/36433/diff/


Testing (updated)
-------

Manually tested

#Local test results: PASSED

#Jenkins test results: 

Running org.apache.ambari.server.controller.internal.ArtifactResourceProviderTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.23 sec

Running org.apache.ambari.server.upgrade.UpgradeCatalog210Test
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.795 sec

Tests run: 3117, Failures: 0, Errors: 0, Skipped: 28

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:20 h
[INFO] Finished at: 2015-07-12T15:37:50+00:00
[INFO] Final Memory: 47M/659M
[INFO] ------------------------------------------------------------------------


Thanks,

Robert Levas


Re: Review Request 36433: kinit of hdfs Kerberos identity fails when starting added service(s) after upgrade to Ambari 2.1.0

Posted by Mahadev Konar <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36433/#review91411
-----------------------------------------------------------

Ship it!


Ship It!

- Mahadev Konar


On July 12, 2015, 2:19 p.m., Robert Levas wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36433/
> -----------------------------------------------------------
> 
> (Updated July 12, 2015, 2:19 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, John Speidel, Mahadev Konar, Robert Nettleton, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-12356
>     https://issues.apache.org/jira/browse/AMBARI-12356
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> STR:
> 1. Install old version of ambari (2.0.1)
> 2. Enable security
> 3. Do Ambari only upgrade to ambari2.1.0
> 4. Add some component - HiveServer2 or Ooozie server
> 5. Try to start added component
> 
> Actual result:
> Start have been failed. 
> 
> ```
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 182, in <module>
>     HiveServer().execute()
>   File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", 
> 
> line 216, in execute
>     method(env)
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 83, in start
>     self.configure(env) # FOR SECURITY
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 54, in configure
>     hive(name='hiveserver2')
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in 
> 
> thunk
>     return fn(*args, **kwargs)
>   File "/var/lib/ambari-agent/cache/common-
> 
> services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 127, in hive
>     mode=params.webhcat_hdfs_user_mode
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 
> 
> __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 390, in 
> 
> action_create_on_execute
>     self.action_delayed("create")
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 387, in 
> 
> action_delayed
>     self.get_hdfs_resource_executor().action_delayed(action_name, self)
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 236, in 
> 
> action_delayed
>     main_resource.kinit()
>   File "/usr/lib/python2.6/site-
> 
> packages/resource_management/libraries/providers/hdfs_resource.py", line 416, in kinit
>     user=user
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in 
> 
> __init__
>     self.env.run()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 152, in run
>     self.run_action(resource, action)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 
> 
> 118, in run_action
>     provider_action()
>   File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", 
> 
> line 254, in action_run
>     tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in 
> 
> inner
>     result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in 
> 
> checked_call
>     tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in 
> 
> _call_wrapper
>     result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in 
> 
> _call
>     raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> 
> /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM' returned 1. kinit: Keytab 
> 
> contains no suitable keys for hdfs@EXAMPLE.COM while getting initial credentials
> ```
> 
> Expected results:
> Can start all added components.
> 
> # Cause
> The Kerberos Descriptor structure changed between Ambari 2.0 and Ambari 2.1.  This change moved the "hdfs" Kerberos identity descriptor from the _global_ scope to under the HDFS service. After upgrading from Ambari 2.0 to Ambari 2.1  an additional "hdfs" Kerberos identity descriptor was added with the new principal name pattern - ${hadoop-env/hdfs_user}-${cluster_name}@${realm}.  This occurred because the stored Kerberos Descriptor contained the _old_ structure, and when Ambari generated a composite Kerberos Descriptor made up of the Kerberos Descriptor compiled from the relevant stack definition with stored changes applied, that additional "hdfs" Kerberos identity descriptor was added.  Because if this, the Kerberos logic became _confused_ and overwrote the existing hdfs keytab file with one that contained the new principal name.
> 
> # Solution
> While migrating Ambari 2.0 to Ambari 2.1, fix the stored Kerberos Descriptor structure to match the new version's structure.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ArtifactResourceProvider.java 680f9b8 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ArtifactDAO.java 27346dd 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ArtifactEntity.java 849a938 
>   ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog210.java 3d4d701 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ArtifactResourceProviderTest.java 789fb54 
>   ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog210Test.java 8708047 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_no_hdfs.json PRE-CREATION 
>   ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_simple.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/36433/diff/
> 
> 
> Testing
> -------
> 
> Manually tested
> 
> #Local test results: PASSED
> 
> #Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>