You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Emil Anca <ea...@hortonworks.com> on 2015/05/08 13:31:24 UTC
Review Request 33974: Kerberos: Keytab files are not distributed
during add host if a retry is necessary during installation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33974/
-----------------------------------------------------------
Review request for Ambari, Robert Levas and Vitalyi Brodetskyi.
Bugs: AMBARI-11022
https://issues.apache.org/jira/browse/AMBARI-11022
Repository: ambari
Description
-------
When adding a new host to a cluster where Kerberos is enabled and the installation of the new components fails, upon retry the keytabs are not distributed to the host after successfully installing the components. Note: the new identities were not created either.
Workaround
To recover from this, the missing keytabs can be regenerated using the Regenerate Keytabs feature with the missing only option specified. The component can then be started successfully.
Steps to reproduce
Create cluster (can be small, one node with only HDFS and Zookeeper)
Enable Kerberos
Add new host with only DataNode (no clients, only to make the failure happen quicker)
While the relevant hadoop packages are being installed, kill the package manger (i.e., yum, zypper, etc...)
The installation of the component will fail and the retry button will be available
Click the retry button and allow the installation to complete
Startup of the Datanode component will fail due to missing keytab
2015-03-21 01:43:47,911 FATAL datanode.DataNode (DataNode.java:secureMain(2385)) - Exception in secureMain
java.io.IOException: Login failure for dn/c6504.ambari.apache.org@EXAMPLE.COM from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
Note: Error indicates a keytab file was found but wrong password, this isn't the case since the keytab file was not on the host.
Problem: If components installation fails and a retry is performed, the Kerberos related component configuration is skipped on a sequential attempts;
Solution: Components transitioning from INSTALL_FAILED->INSTALLED state should also be taken into account.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 7b77bfa
Diff: https://reviews.apache.org/r/33974/diff/
Testing
-------
mvn clean test -pl ambari-server
Total run:765
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47:47.894s
[INFO] Finished at: Thu May 07 19:13:42 EEST 2015
[INFO] Final Memory: 47M/507M
[INFO] ------------------------------------------------------------------------
Thanks,
Emil Anca
Re: Review Request 33974: Kerberos: Keytab files are not distributed
during add host if a retry is necessary during installation
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33974/#review82995
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On May 8, 2015, 7:31 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33974/
> -----------------------------------------------------------
>
> (Updated May 8, 2015, 7:31 a.m.)
>
>
> Review request for Ambari, Robert Levas and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-11022
> https://issues.apache.org/jira/browse/AMBARI-11022
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When adding a new host to a cluster where Kerberos is enabled and the installation of the new components fails, upon retry the keytabs are not distributed to the host after successfully installing the components. Note: the new identities were not created either.
> Workaround
> To recover from this, the missing keytabs can be regenerated using the Regenerate Keytabs feature with the missing only option specified. The component can then be started successfully.
> Steps to reproduce
> Create cluster (can be small, one node with only HDFS and Zookeeper)
> Enable Kerberos
> Add new host with only DataNode (no clients, only to make the failure happen quicker)
> While the relevant hadoop packages are being installed, kill the package manger (i.e., yum, zypper, etc...)
> The installation of the component will fail and the retry button will be available
> Click the retry button and allow the installation to complete
> Startup of the Datanode component will fail due to missing keytab
> 2015-03-21 01:43:47,911 FATAL datanode.DataNode (DataNode.java:secureMain(2385)) - Exception in secureMain
> java.io.IOException: Login failure for dn/c6504.ambari.apache.org@EXAMPLE.COM from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
> Note: Error indicates a keytab file was found but wrong password, this isn't the case since the keytab file was not on the host.
>
>
> Problem: If components installation fails and a retry is performed, the Kerberos related component configuration is skipped on a sequential attempts;
> Solution: Components transitioning from INSTALL_FAILED->INSTALLED state should also be taken into account.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 7b77bfa
>
> Diff: https://reviews.apache.org/r/33974/diff/
>
>
> Testing
> -------
>
> mvn clean test -pl ambari-server
>
> Total run:765
> Total errors:0
> Total failures:0
> OK
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 47:47.894s
> [INFO] Finished at: Thu May 07 19:13:42 EEST 2015
> [INFO] Final Memory: 47M/507M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>