You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by John Speidel <js...@hortonworks.com> on 2015/04/01 01:31:05 UTC
Re: Review Request 32604: Cannot enable kerberos with Ambari server
running non-root
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32604/#review78430
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
<https://reviews.apache.org/r/32604/#comment127113>
I know that this behavior wasn't changed in this patch but I have a question about how a failure to set permissions on a file is handled. My first impression is that it should result in an exception being propagated but currently only results in a log msg. What is the rational to this not being treated as an exception? In addition to ambari not being able to read/write files due to a failure, the presumably more dangerous situation would be that there is a failure to lock down permissions and everyone has access to keytabs, etc. which may not be noticed if ambari is working properly.
- John Speidel
On March 31, 2015, 11:04 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32604/
> -----------------------------------------------------------
>
> (Updated March 31, 2015, 11:04 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly, John Speidel, Mahadev Konar, and Yusaku Sako.
>
>
> Bugs: AMBARI-10266
> https://issues.apache.org/jira/browse/AMBARI-10266
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2, Ambari 2.0, MIT KDC, during setup, set to use a sudoer account for ambari-server. start ambari server.
>
> failing during "Test Kerberos Client", see attached.
>
> ```
> 27 Mar 2015 17:55:46,449 ERROR [qtp-client-19] KerberosHelper:1330 - Failed to create the temporary data directory.
> java.io.IOException: Failed to create a temporary directory in /var/lib/ambari-server/tmp
> at org.apache.ambari.server.controller.KerberosHelper.createTemporaryDirectory(KerberosHelper.java:1324)
> at org.apache.ambari.server.controller.KerberosHelper.handleTestIdentity(KerberosHelper.java:945)
> at org.apache.ambari.server.controller.KerberosHelper.createTestIdentity(KerberosHelper.java:433)
> at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImpl.java:2944)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:161)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:158)
> at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider.java:158)
> at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
> at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:75)
> at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
> at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
> at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
> at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:103)
> at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:72)
> at org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:145)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:178)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
> at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at org.apache.ambari.server.controller.FailsafeHandlerList.handleNonFailSafe(FailsafeHandlerList.java:148)
> at org.apache.ambari.server.controller.AmbariHandlerList.handleNonFailSafe(AmbariHandlerList.java:157)
> at org.apache.ambari.server.controller.FailsafeHandlerList.handle(FailsafeHandlerList.java:130)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:363)
> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:931)
> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:992)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> ```
>
> This is due to an incorrect temporary directory in the ambari.properties file. Once the (incorrect) temporary directory was manually created. The following error is observed:
>
> ```
> 27 Mar 2015 18:00:10,630 INFO [Server Action Executor Worker 55] CreateKeytabFilesServerAction:170 - Creating keytab file for ambari-qa_mngxikyl@EXAMPLE.COM on host ip-10-99-165-62.ec2.internal
> 27 Mar 2015 18:00:10,755 ERROR [Server Action Executor Worker 55] KerberosOperationHandler:433 - Failed to export keytab file
> java.io.FileNotFoundException: /var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/1ce5855f40583660938abde762237b442c2d5ba9 (Permission denied)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.writeFile(Keytab.java:273)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.write(Keytab.java:133)
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:429)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.processIdentity(CreateKeytabFilesServerAction.java:276)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:494)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:386)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.execute(CreateKeytabFilesServerAction.java:99)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:504)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:441)
> at java.lang.Thread.run(Thread.java:745)
> 27 Mar 2015 18:00:10,756 ERROR [Server Action Executor Worker 55] CreateKeytabFilesServerAction:290 - Failed to create keytab file for ambari-qa_mngxikyl@EXAMPLE.COM - Failed to export keytab file
> org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Failed to export keytab file
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:439)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.processIdentity(CreateKeytabFilesServerAction.java:276)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:494)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:386)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.execute(CreateKeytabFilesServerAction.java:99)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:504)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:441)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: /var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/1ce5855f40583660938abde762237b442c2d5ba9 (Permission denied)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.writeFile(Keytab.java:273)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.write(Keytab.java:133)
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:429)
> ... 7 more
> 27 Mar 2015 18:00:10,757 INFO
> ```
>
> This was caused by incorrect permissions on `/var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/` where the directory was set to 600 (rw-------) rather than 700 (rwx------).
>
> **Solution**
> - Fix ambari.properties to have the correct `server.tmp.dir` value, which is `/var/lib/ambari-server/data/tmp`
> - Make sure the directories in the created path has executable permissions for the user that Ambari executes as
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 44aea5c
> ambari-server/conf/windows/ambari.properties cfe9c3d
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java 3e94cd6
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerActionTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/32604/diff/
>
>
> Testing
> -------
>
> Manual testing in cluster to ensure directory permissions were set properly.
> Addd new unit test to ensure directory and file permissions were set properly in `org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction#ensureAmbariOnlyAccess`
>
> **Local test results:**
> Running org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerActionTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 sec
>
> Tests run: 2804, Failures: 0, Errors: 0, Skipped: 15
>
> **Jenkins test results:**
> Failed due to unrelated issues:
> ```
> [ERROR] Failed to execute goal on project ambari-server: Could not resolve dependencies for project org.apache.ambari:ambari-server:jar:2.0.0-SNAPSHOT: Failed to collect dependencies at org.apache.ambari:ambari-metrics-common:jar:2.0.0-SNAPSHOT: Failed to read artifact descriptor for org.apache.ambari:ambari-metrics-common:jar:2.0.0-SNAPSHOT: Could not find artifact org.apache.ambari:ambari-metrics:pom:2.0.0-SNAPSHOT in oss.sonatype.org (https://oss.sonatype.org/content/groups/staging) -> [Help 1]
> ```
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 32604: Cannot enable kerberos with Ambari server
running non-root
Posted by Robert Levas <rl...@hortonworks.com>.
> On March 31, 2015, 7:31 p.m., John Speidel wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java, line 377
> > <https://reviews.apache.org/r/32604/diff/1/?file=908959#file908959line377>
> >
> > I know that this behavior wasn't changed in this patch but I have a question about how a failure to set permissions on a file is handled. My first impression is that it should result in an exception being propagated but currently only results in a log msg. What is the rational to this not being treated as an exception? In addition to ambari not being able to read/write files due to a failure, the presumably more dangerous situation would be that there is a failure to lock down permissions and everyone has access to keytabs, etc. which may not be noticed if ambari is working properly.
When created, root was only considered (bad planning on my part). Therefore Ambari's access to the file wasn't an issue. This method was meant to only suggest ALCs for group and other access.
I will change the logging of the message to throwing an exception.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32604/#review78430
-----------------------------------------------------------
On March 31, 2015, 7:04 p.m., Robert Levas wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32604/
> -----------------------------------------------------------
>
> (Updated March 31, 2015, 7:04 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly, John Speidel, Mahadev Konar, and Yusaku Sako.
>
>
> Bugs: AMBARI-10266
> https://issues.apache.org/jira/browse/AMBARI-10266
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HDP 2.2, Ambari 2.0, MIT KDC, during setup, set to use a sudoer account for ambari-server. start ambari server.
>
> failing during "Test Kerberos Client", see attached.
>
> ```
> 27 Mar 2015 17:55:46,449 ERROR [qtp-client-19] KerberosHelper:1330 - Failed to create the temporary data directory.
> java.io.IOException: Failed to create a temporary directory in /var/lib/ambari-server/tmp
> at org.apache.ambari.server.controller.KerberosHelper.createTemporaryDirectory(KerberosHelper.java:1324)
> at org.apache.ambari.server.controller.KerberosHelper.handleTestIdentity(KerberosHelper.java:945)
> at org.apache.ambari.server.controller.KerberosHelper.createTestIdentity(KerberosHelper.java:433)
> at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImpl.java:2944)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:161)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:158)
> at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
> at org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider.java:158)
> at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
> at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:75)
> at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
> at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
> at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
> at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:103)
> at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:72)
> at org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:145)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:178)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
> at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at org.apache.ambari.server.controller.FailsafeHandlerList.handleNonFailSafe(FailsafeHandlerList.java:148)
> at org.apache.ambari.server.controller.AmbariHandlerList.handleNonFailSafe(AmbariHandlerList.java:157)
> at org.apache.ambari.server.controller.FailsafeHandlerList.handle(FailsafeHandlerList.java:130)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:363)
> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)
> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:931)
> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:992)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> ```
>
> This is due to an incorrect temporary directory in the ambari.properties file. Once the (incorrect) temporary directory was manually created. The following error is observed:
>
> ```
> 27 Mar 2015 18:00:10,630 INFO [Server Action Executor Worker 55] CreateKeytabFilesServerAction:170 - Creating keytab file for ambari-qa_mngxikyl@EXAMPLE.COM on host ip-10-99-165-62.ec2.internal
> 27 Mar 2015 18:00:10,755 ERROR [Server Action Executor Worker 55] KerberosOperationHandler:433 - Failed to export keytab file
> java.io.FileNotFoundException: /var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/1ce5855f40583660938abde762237b442c2d5ba9 (Permission denied)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.writeFile(Keytab.java:273)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.write(Keytab.java:133)
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:429)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.processIdentity(CreateKeytabFilesServerAction.java:276)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:494)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:386)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.execute(CreateKeytabFilesServerAction.java:99)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:504)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:441)
> at java.lang.Thread.run(Thread.java:745)
> 27 Mar 2015 18:00:10,756 ERROR [Server Action Executor Worker 55] CreateKeytabFilesServerAction:290 - Failed to create keytab file for ambari-qa_mngxikyl@EXAMPLE.COM - Failed to export keytab file
> org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Failed to export keytab file
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:439)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.processIdentity(CreateKeytabFilesServerAction.java:276)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processRecord(KerberosServerAction.java:494)
> at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:386)
> at org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction.execute(CreateKeytabFilesServerAction.java:99)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:504)
> at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:441)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: /var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/1ce5855f40583660938abde762237b442c2d5ba9 (Permission denied)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.writeFile(Keytab.java:273)
> at org.apache.directory.server.kerberos.shared.keytab.Keytab.write(Keytab.java:133)
> at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.createKeytabFile(KerberosOperationHandler.java:429)
> ... 7 more
> 27 Mar 2015 18:00:10,757 INFO
> ```
>
> This was caused by incorrect permissions on `/var/lib/ambari-server/tmp/.ambari_1427493565210-0.d/ip-10-99-165-62.ec2.internal/` where the directory was set to 600 (rw-------) rather than 700 (rwx------).
>
> **Solution**
> - Fix ambari.properties to have the correct `server.tmp.dir` value, which is `/var/lib/ambari-server/data/tmp`
> - Make sure the directories in the created path has executable permissions for the user that Ambari executes as
>
>
> Diffs
> -----
>
> ambari-server/conf/unix/ambari.properties 44aea5c
> ambari-server/conf/windows/ambari.properties cfe9c3d
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java 3e94cd6
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerActionTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/32604/diff/
>
>
> Testing
> -------
>
> Manual testing in cluster to ensure directory permissions were set properly.
> Addd new unit test to ensure directory and file permissions were set properly in `org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerAction#ensureAmbariOnlyAccess`
>
> **Local test results:**
> Running org.apache.ambari.server.serveraction.kerberos.CreateKeytabFilesServerActionTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 sec
>
> Tests run: 2804, Failures: 0, Errors: 0, Skipped: 15
>
> **Jenkins test results:**
> Failed due to unrelated issues:
> ```
> [ERROR] Failed to execute goal on project ambari-server: Could not resolve dependencies for project org.apache.ambari:ambari-server:jar:2.0.0-SNAPSHOT: Failed to collect dependencies at org.apache.ambari:ambari-metrics-common:jar:2.0.0-SNAPSHOT: Failed to read artifact descriptor for org.apache.ambari:ambari-metrics-common:jar:2.0.0-SNAPSHOT: Could not find artifact org.apache.ambari:ambari-metrics:pom:2.0.0-SNAPSHOT in oss.sonatype.org (https://oss.sonatype.org/content/groups/staging) -> [Help 1]
> ```
>
>
> Thanks,
>
> Robert Levas
>
>