You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Levas <rl...@hortonworks.com> on 2015/02/24 19:24:17 UTC
Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
Bugs: AMBARI-9775
https://issues.apache.org/jira/browse/AMBARI-9775
Repository: ambari
Description
-------
On 2.0 and 2.1 stack oozie server failed with following error:
```
2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
at org.apache.oozie.service.Services.setService(Services.java:358)
at org.apache.oozie.service.Services.loadServices(Services.java:291)
at org.apache.oozie.service.Services.init(Services.java:212)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
... 31 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
... 32 more
2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
```
Solution: replace _HOST in principal names for relevant Oozie versions.
Diffs
-----
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
Diff: https://reviews.apache.org/r/31368/diff/
Testing
-------
unit tests passed, service checks passed on all stacks
# Jenkins test results: PENDING
Thanks,
Robert Levas
Re: Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
Posted by Eugene Chekanskiy <ec...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/#review73914
-----------------------------------------------------------
Ship it!
Ship It!
- Eugene Chekanskiy
On Feb. 24, 2015, 6:24 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31368/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 6:24 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9775
> https://issues.apache.org/jira/browse/AMBARI-9775
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On 2.0 and 2.1 stack oozie server failed with following error:
> ```
> 2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
> at org.apache.oozie.service.Services.setService(Services.java:358)
> at org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
> ... 31 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>
> at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
> ... 32 more
> 2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
> ```
>
> Solution: replace _HOST in principal names for relevant Oozie versions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
>
> Diff: https://reviews.apache.org/r/31368/diff/
>
>
> Testing
> -------
>
> unit tests passed, service checks passed on all stacks
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/#review74027
-----------------------------------------------------------
Ship it!
Ship It!
- Andrew Onischuk
On Feb. 24, 2015, 6:24 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31368/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 6:24 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9775
> https://issues.apache.org/jira/browse/AMBARI-9775
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On 2.0 and 2.1 stack oozie server failed with following error:
> ```
> 2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
> at org.apache.oozie.service.Services.setService(Services.java:358)
> at org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
> ... 31 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>
> at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
> ... 32 more
> 2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
> ```
>
> Solution: replace _HOST in principal names for relevant Oozie versions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
>
> Diff: https://reviews.apache.org/r/31368/diff/
>
>
> Testing
> -------
>
> unit tests passed, service checks passed on all stacks
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 24, 2015, 2:25 p.m., Andrew Onischuk wrote:
> > This approach is so hacky, when we need substitution we ussually send templates like {{hostname}}, and then they get automatically replaced. This what we do for all our substitutions.
>
> Robert Levas wrote:
> I don't think this is _hacky_. We can not use the `{{hostnanme}}` approach since that is a Python-specific construct and the princial name is being generated by Ambari. Therefore we are left to do the explicit replacement on the agent-side using `.replace('_HOST', hostname)`. This method is done throughout the code.
>
> I believe that altering the immutable dictionary is not a good idea, but the relevant set of data should be copied to a new dictionary and modified there.
Correction: the origianl dictionaly is not changed, I copy is being created. I think this is the correct approach.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/#review73872
-----------------------------------------------------------
On Feb. 24, 2015, 1:24 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31368/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 1:24 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9775
> https://issues.apache.org/jira/browse/AMBARI-9775
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On 2.0 and 2.1 stack oozie server failed with following error:
> ```
> 2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
> at org.apache.oozie.service.Services.setService(Services.java:358)
> at org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
> ... 31 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>
> at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
> ... 32 more
> 2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
> ```
>
> Solution: replace _HOST in principal names for relevant Oozie versions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
>
> Diff: https://reviews.apache.org/r/31368/diff/
>
>
> Testing
> -------
>
> unit tests passed, service checks passed on all stacks
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
Posted by Robert Levas <rl...@hortonworks.com>.
> On Feb. 24, 2015, 2:25 p.m., Andrew Onischuk wrote:
> > This approach is so hacky, when we need substitution we ussually send templates like {{hostname}}, and then they get automatically replaced. This what we do for all our substitutions.
I don't think this is _hacky_. We can not use the `{{hostnanme}}` approach since that is a Python-specific construct and the princial name is being generated by Ambari. Therefore we are left to do the explicit replacement on the agent-side using `.replace('_HOST', hostname)`. This method is done throughout the code.
I believe that altering the immutable dictionary is not a good idea, but the relevant set of data should be copied to a new dictionary and modified there.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/#review73872
-----------------------------------------------------------
On Feb. 24, 2015, 1:24 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31368/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 1:24 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9775
> https://issues.apache.org/jira/browse/AMBARI-9775
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On 2.0 and 2.1 stack oozie server failed with following error:
> ```
> 2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
> at org.apache.oozie.service.Services.setService(Services.java:358)
> at org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
> ... 31 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>
> at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
> ... 32 more
> 2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
> ```
>
> Solution: replace _HOST in principal names for relevant Oozie versions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
>
> Diff: https://reviews.apache.org/r/31368/diff/
>
>
> Testing
> -------
>
> unit tests passed, service checks passed on all stacks
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 31368: Oozie failed to start in secured cluster for
stacks 2.0 and 2.1
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31368/#review73872
-----------------------------------------------------------
This approach is so hacky, when we need substitution we ussually send templates like {{hostname}}, and then they get automatically replaced. This what we do for all our substitutions.
- Andrew Onischuk
On Feb. 24, 2015, 6:24 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31368/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 6:24 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Eugene Chekanskiy, and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9775
> https://issues.apache.org/jira/browse/AMBARI-9775
>
>
> Repository: ambari
>
>
> Description
> -------
>
> On 2.0 and 2.1 stack oozie server failed with following error:
> ```
> 2015-02-23 16:23:54,474 WARN NativeCodeLoader:62 - SERVER[] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-02-23 16:23:54,821 FATAL Services:533 - SERVER[] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
> at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:372)
> at org.apache.oozie.service.Services.setService(Services.java:358)
> at org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:212)
> at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> Caused by: java.io.IOException: Login failure for oozie/_HOST@EXAMPLE.COM from keytab /etc/security/keytabs/oozie.service.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:870)
> at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
> ... 31 more
> Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
>
> at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:861)
> ... 32 more
> 2015-02-23 16:23:54,833 INFO Services:539 - SERVER[] Shutdown
> ```
>
> Solution: replace _HOST in principal names for relevant Oozie versions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2065f4a
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py efe11f5
>
> Diff: https://reviews.apache.org/r/31368/diff/
>
>
> Testing
> -------
>
> unit tests passed, service checks passed on all stacks
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>