Re: regression in 2.4? YARN severs on secure cluster startup

[Steve Loughran <st...@hortonworks.com>]

I can confirm that YARN-1600 fixes the YARN problem, so you can upgrade the
YARN binaries without having to patch the yarn-site.xml to get them to
start again.

However, that leaves the need to set the
'dfs.web.authentication.kerberos.principal'
on a secure cluster once you've upgrade -if you hadn't already.

https://issues.apache.org/jira/browse/HDFS-5532 "Enable the webhdfs by
default to support new HDFS web UI"

I think that makes it something to pull up as potentially incompatible,
with the note in CHANGES.TXT -you need to set  'dfs.web.authentication.
kerberos.principal on a secure cluster.

I've stuck my stack trace into the JIRA so it will be searchable


On 29 January 2014 17:13, Jason Lowe <jl...@yahoo-inc.com> wrote:

> The RM issue should be fixed by YARN-1600 which I just committed this
> morning.
>
> Jason
>
>
> On 01/29/2014 10:33 AM, Steve Loughran wrote:
>
>> I'm just switching over to use the 2.4-SNAPSHOT in a secured pseudo-dist
>> cluster, and now the services are failing to come up because the web
>> principals haven't been defined. Example
>>
>>
>> 2014-01-29 15:42:58,558 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter (class=org.apache.hadoop.ht
>> tp.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
>> 2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter (class=org.apache.hadoop.ht
>> tp.lib.StaticUserWebFilter$StaticUserFilter) to context static
>> 2014-01-29 15:42:58,559 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter static_user_filter (class=org.apache.hadoop.ht
>> tp.lib.StaticUserWebFilter$StaticUserFilter) to context logs
>> 2014-01-29 15:42:58,630 ERROR org.apache.hadoop.http.HttpServer2:
>> *WebHDFS
>> and security are enabled, but configuration proper*
>> *ty 'dfs.web.authentication.kerberos.principal' is not set.*
>>
>> 2014-01-29 15:42:58,630 INFO org.apache.hadoop.http.HttpServer2: Added
>> filter 'SPNEGO' (class=org.apache.hadoop.hdfs.web.Aut
>> hFilter)
>> 2014-01-29 15:42:58,631 INFO org.apache.hadoop.http.HttpServer2:
>> addJerseyResourcePackage: packageName=org.apache.hadoop.hdf
>> s.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
>> pathSpec=/webhdfs/v1/*
>> 2014-01-29 15:42:58,658 INFO org.apache.hadoop.http.HttpServer2: Adding
>> Kerberos (SPNEGO) filter to getDelegationToken
>> 2014-01-29 15:42:58,662 INFO org.apache.hadoop.http.HttpServer2: Adding
>> Kerberos (SPNEGO) filter to renewDelegationToken
>> 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
>> Kerberos (SPNEGO) filter to cancelDelegationToken
>> 2014-01-29 15:42:58,663 INFO org.apache.hadoop.http.HttpServer2: Adding
>> Kerberos (SPNEGO) filter to fsck
>> 2014-01-29 15:42:58,671 INFO org.apache.hadoop.http.HttpServer2: Adding
>> Kerberos (SPNEGO) filter to getimage
>> 2014-01-29 15:42:58,748 INFO org.apache.hadoop.http.HttpServer2: Jetty
>> bound to port 50070
>> 2014-01-29 15:42:58,748 INFO org.mortbay.log: jetty-6.1.26
>> 2014-01-29 15:42:58,941 INFO
>> org.apache.hadoop.security.authentication.server.
>> KerberosAuthenticationHandler:
>> Login using key
>> tab /home/stevel/conf/hdfs.keytab, for principal HTTP/ubuntu@COTHAM
>> 2014-01-29 15:42:58,981 INFO
>> org.apache.hadoop.security.authentication.server.
>> KerberosAuthenticationHandler:
>> Initialized, pr
>> incipal [HTTP/ubuntu@COTHAM] from keytab [/home/stevel/conf/hdfs.keytab]
>> 2014-01-29 15:42:58,981 WARN
>> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
>> 'signature.secret' confi
>> guration not set, using a random value as secret
>> 2014-01-29 15:42:58,982 WARN org.mortbay.log: failed SPNEGO:
>> javax.servlet.ServletException: javax.servlet.ServletException:
>>   Principal not defined in configuration
>> 2014-01-29 15:42:58,982 WARN org.mortbay.log: Failed startup of context
>> org.mortbay.jetty.webapp.WebAppContext@167a465{/,fil
>> e:/home/stevel/hadoop/share/hadoop/hdfs/webapps/hdfs}
>> javax.servlet.ServletException: javax.servlet.ServletException: Principal
>> not defined in configuration
>>          at
>> org.apache.hadoop.security.authentication.server.
>> KerberosAuthenticationHandler.init(KerberosAuthenticationHandler
>> .java:203)
>> :
>>
>> YARN is the same but without the text telling me what config option I have
>> to set (i.e no equivalent of https://issues.apache.org/
>> jira/browse/HDFS-3813)
>>
>> -29 16:04:33,908 INFO
>> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
>> AMLivelinessMonitor
>> thread interrupted
>> 2014-01-29 16:04:33,908 INFO
>> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.
>> ContainerAllocationExpirer
>> thread interrupted
>> 2014-01-29 16:04:33,908 ERROR
>> org.apache.hadoop.security.token.delegation.
>> AbstractDelegationTokenSecretManager:
>> InterruptedExcpetion recieved for ExpiredTokenRemover thread
>> java.lang.InterruptedException: sleep interrupted
>> 2014-01-29 16:04:33,909 INFO
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
>> Transitioned
>> to standby state
>> 2014-01-29 16:04:33,909 FATAL
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
>> starting ResourceManager
>> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
>> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:250)
>> at
>> org.apache.hadoop.yarn.server.resourcemanager.
>> ResourceManager.startWepApp(ResourceManager.java:775)
>> at
>> org.apache.hadoop.yarn.server.resourcemanager.
>> ResourceManager.serviceStart(ResourceManager.java:866)
>> at org.apache.hadoop.service.AbstractService.start(
>> AbstractService.java:193)
>> at
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(
>> ResourceManager.java:995)
>> Caused by: java.io.IOException: Unable to initialize WebAppContext
>> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
>> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:245)
>> ... 4 more
>> Caused by: javax.servlet.ServletException: javax.servlet.
>> ServletException:
>> Principal not defined in configuration
>> at
>> org.apache.hadoop.security.authentication.server.
>> KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.
>> java:203)
>> at
>> org.apache.hadoop.security.authentication.server.
>> AuthenticationFilter.init(AuthenticationFilter.java:146)
>> at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
>> at org.mortbay.component.AbstractLifeCycle.start(
>> AbstractLifeCycle.java:50)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.initialize(
>> ServletHandler.java:713)
>> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
>> at
>> org.mortbay.jetty.webapp.WebAppContext.startContext(
>> WebAppContext.java:1282)
>> at org.mortbay.jetty.handler.ContextHandler.doStart(
>> ContextHandler.java:518)
>> at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
>> at org.mortbay.component.AbstractLifeCycle.start(
>> AbstractLifeCycle.java:50)
>> at
>> org.mortbay.jetty.handler.HandlerCollection.doStart(
>> HandlerCollection.java:152)
>> at
>> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(
>> ContextHandlerCollection.java:156)
>> at org.mortbay.component.AbstractLifeCycle.start(
>> AbstractLifeCycle.java:50)
>> at org.mortbay.jetty.handler.HandlerWrapper.doStart(
>> HandlerWrapper.java:130)
>> at org.mortbay.jetty.Server.doStart(Server.java:224)
>> at org.mortbay.component.AbstractLifeCycle.start(
>> AbstractLifeCycle.java:50)
>> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:787)
>> ... 5 more
>> Caused by: javax.servlet.ServletException: Principal not defined in
>> configuration
>> at
>> org.apache.hadoop.security.authentication.server.
>> KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.
>> java:164)
>> ... 21 more
>>
>> This cluster config (
>> https://github.com/hortonworks/hoya/tree/master/
>> hoya-funtest/src/test/configs/ubuntu-secure)
>> did work on Hadoop-2.2, which implies that there is some change in
>> Hadoop
>> 2.4 that is incompatible to the extent that you now need to declare some
>> principals you didn't need to set now need to be
>>
>>
>> 1. it Looks like HDFS-3604, enable webhdfs is the issue that stops the NN
>> and DN  -IMO it should be marked as an incompatible change, as it stops
>> clusters that used to start, from starting.
>> 2. what's causing this in YARN?
>>
>>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.