You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Kevin Buckley <ke...@gmail.com> on 2017/07/20 06:34:56 UTC
Kerberised JobHistory Server not starting: User jhs trying to create
the /mr-history/done directory
My Hadoop 2.8.0's
/mr-history/done
directory is owned by the mapred user, who is in the hadoop group,
and the directory has the pemissions
/mr-history":mapred:hadoop:drwxrwx---
If I run the Hadoop instance without any Kerberos config, and
fire up the JobHistory server as the mapred user, everything
works.
If I flip over to a Kerberised environment, the NameNode and DataNodes,
running as the 'hdfs' user, and the Resource and and Node Managers, running
as the 'yarn' user, all start up OK and their respective web exposure can be
used.
When I try to start up the JobHistory server however
/bin/su mapred -c
'/local/Hadoop/hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh --config
/local/Hadoop/hadoop-2.8.0/etc/hadoop/ start historyserver
I get a message in the logs telling me that, rather than the mapred
user doing things,
a user 'jhs' is trying to do stuff, vis
2017-07-20 18:15:09,667 INFO
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX
signal handlers for [TERM, HUP, INT]
2017-07-20 18:15:10,062 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for
user jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ using keytab file
/local/Hadoop/krb/jhs.service.keytab
2017-07-20 18:15:10,107 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2017-07-20 18:15:10,142 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2017-07-20 18:15:10,142 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer
metrics system started
2017-07-20 18:15:10,145 INFO
org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
2017-07-20 18:15:10,411 INFO
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default
file system [hdfs://co246a-a.ecs.vuw.ac.nz:9000]
2017-07-20 18:15:10,518 INFO
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state
INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
Error creating done directory:
[hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating
done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:639)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:585)
at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:550)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:95)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:151)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241)
Caused by: org.apache.hadoop.security.AccessControlException:
Permission denied: user=jhs, access=EXECUTE,
inode="/mr-history":mapred:hadoop:drwxrwx---
But where has the jhs user come from ?
Doesn't appear to be set anywhere in any of the config files.
According to the hadoop-2.8.0 docs SecureMode page,
https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/SecureMode.html
=============================================
MapReduce JobHistory Server
The MapReduce JobHistory Server keytab file, on that host, should look
like the following:
$ klist -e -k -t /etc/security/keytab/jhs.service.keytab
Keytab name: FILE:/etc/security/keytab/jhs.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
(AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
(AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
(ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
(AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
(AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
(ArcFour with HMAC/md5)
=============================================
and mine does.
The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
play around with the
hadoop.security.auth_to_local
config value, but I haven't had to do that for the nn, dn, rm or nm keytabs.
So is there something special about the jhs user ?
Or perhaps something special about the other keytab values ?
Any clues/insight welcome,
Kevin
---
Kevin M. Buckley
eScience Consultant
School of Engineering and Computer Science
Victoria University of Wellington
New Zealand
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to
create the /mr-history/done directory
Posted by Kevin Buckley <ke...@gmail.com>.
On 25 July 2017 at 03:21, Erik Krogen <ek...@linkedin.com> wrote:
> Hey Kevin,
>
> Sorry, I missed your point about using auth_to_local. You're right that you should be able to use that for what you're trying to achieve. I think it's just that your rule is wrong; I believe it should be:
>
> RULE:[2:$1@$0](jhs/.*@ECS.VUW.AC.NZ)s/.*/mapred/
>
> HTH
If I ever get a chance to retrace my steps, Erik, then I'll give it a go.
As things stand I went with the altered username in the keytab and
things seem to be working, although I have one other issue that I'm
about to start a new thread for.
Cheers again,
Kevin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to
create the /mr-history/done directory
Posted by Erik Krogen <ek...@linkedin.com>.
Hey Kevin,
Sorry, I missed your point about using auth_to_local. You're right that you should be able to use that for what you're trying to achieve. I think it's just that your rule is wrong; I believe it should be:
RULE:[2:$1@$0](jhs/.*@ECS.VUW.AC.NZ)s/.*/mapred/
HTH
Erik
On 7/23/17, 6:37 PM, "Kevin Buckley" <ke...@gmail.com> wrote:
On 21 July 2017 at 13:25, Kevin Buckley
<ke...@gmail.com> wrote:
> On 21 July 2017 at 04:04, Erik Krogen <ek...@linkedin.com> wrote:
>> Hi Kevin,
>>
>> Since you are using the "jhs" keytab with principal "jhs/_HOST@REALM.TLD",
>> the JHS is authenticating itself as the jhs user (which is the actual
>> important part, rather than the user the process is running as). If you want
>> it to be the "mapred" user, you should change the keytab/principal you use
>> (mapred.jobhistory.{principal,keytab}).
>
> I'll certainly give that a go Erik, however, the way I read the
>
>>> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
>>> play around with the
>>>
>>> hadoop.security.auth_to_local
>
> bits suggested to me that if you set things up such that
>
> =======
> $ hadoop org.apache.hadoop.security.HadoopKerberosName
> jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
> 17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
> mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ after auth_to_local rule
> RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
> Name: jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ to
> mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
> ====
>
> (or even used a rule that just mapped the principal to a simple "mapred"
> because I tried that too !) told you it was remapping the user, then it would
> remap for all instances of the user, within the Hadoop instance..
>
> Let's see.
OK,
so it would appear that despite the Hadoop docs appearing to suggest that
you only need the three usernames, 'hdfs', 'yarn' and 'mapred'. if you do use
the principal from the docs, which has the jhs component, then even if you
do try to map users using 'hadoop.security.auth_to_local', your JobHistory
server will start up, inside Hadoop running as a 'jhs' user.
That would seem to be a bit of a trap for the unaware/unwary that the docs
could easily improve upon ?
Thanks again for the pointer to the correct interpreation of the docs, Erik,
Kevin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to
create the /mr-history/done directory
Posted by Kevin Buckley <ke...@gmail.com>.
On 21 July 2017 at 13:25, Kevin Buckley
<ke...@gmail.com> wrote:
> On 21 July 2017 at 04:04, Erik Krogen <ek...@linkedin.com> wrote:
>> Hi Kevin,
>>
>> Since you are using the "jhs" keytab with principal "jhs/_HOST@REALM.TLD",
>> the JHS is authenticating itself as the jhs user (which is the actual
>> important part, rather than the user the process is running as). If you want
>> it to be the "mapred" user, you should change the keytab/principal you use
>> (mapred.jobhistory.{principal,keytab}).
>
> I'll certainly give that a go Erik, however, the way I read the
>
>>> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
>>> play around with the
>>>
>>> hadoop.security.auth_to_local
>
> bits suggested to me that if you set things up such that
>
> =======
> $ hadoop org.apache.hadoop.security.HadoopKerberosName
> jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
> 17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
> mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ after auth_to_local rule
> RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
> Name: jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ to
> mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
> ====
>
> (or even used a rule that just mapped the principal to a simple "mapred"
> because I tried that too !) told you it was remapping the user, then it would
> remap for all instances of the user, within the Hadoop instance..
>
> Let's see.
OK,
so it would appear that despite the Hadoop docs appearing to suggest that
you only need the three usernames, 'hdfs', 'yarn' and 'mapred'. if you do use
the principal from the docs, which has the jhs component, then even if you
do try to map users using 'hadoop.security.auth_to_local', your JobHistory
server will start up, inside Hadoop running as a 'jhs' user.
That would seem to be a bit of a trap for the unaware/unwary that the docs
could easily improve upon ?
Thanks again for the pointer to the correct interpreation of the docs, Erik,
Kevin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to
create the /mr-history/done directory
Posted by Kevin Buckley <ke...@gmail.com>.
On 21 July 2017 at 04:04, Erik Krogen <ek...@linkedin.com> wrote:
> Hi Kevin,
>
> Since you are using the "jhs" keytab with principal "jhs/_HOST@REALM.TLD",
> the JHS is authenticating itself as the jhs user (which is the actual
> important part, rather than the user the process is running as). If you want
> it to be the "mapred" user, you should change the keytab/principal you use
> (mapred.jobhistory.{principal,keytab}).
I'll certainly give that a go Erik, however, the way I read the
>> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
>> play around with the
>>
>> hadoop.security.auth_to_local
bits suggested to me that if you set things up such that
=======
$ hadoop org.apache.hadoop.security.HadoopKerberosName
jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
17/07/20 17:42:50 INFO util.KerberosName: Non-simple name
mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ after auth_to_local rule
RULE:[2:$1/$2@$0](jhs/.*)s/jhs/mapred/
Name: jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ to
mapred/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ
====
(or even used a rule that just mapped the principal to a simple "mapred"
because I tried that too !) told you it was remapping the user, then it would
remap for all instances of the user, within the Hadoop instance..
Let's see.
Cheers again for the feedback.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Kerberised JobHistory Server not starting: User jhs trying to
create the /mr-history/done directory
Posted by Erik Krogen <ek...@linkedin.com.INVALID>.
Hi Kevin,
Since you are using the "jhs" keytab with principal "jhs/_HOST@REALM.TLD",
the JHS is authenticating itself as the jhs user (which is the actual
important part, rather than the user the process is running as). If you
want it to be the "mapred" user, you should change the keytab/principal you
use (mapred.jobhistory.{principal,keytab}).
HTH,
Erik
On Wed, Jul 19, 2017 at 11:34 PM, Kevin Buckley <
kevin.buckley.ecs.vuw.ac.nz@gmail.com> wrote:
> My Hadoop 2.8.0's
>
> /mr-history/done
>
> directory is owned by the mapred user, who is in the hadoop group,
> and the directory has the pemissions
>
> /mr-history":mapred:hadoop:drwxrwx---
>
> If I run the Hadoop instance without any Kerberos config, and
> fire up the JobHistory server as the mapred user, everything
> works.
>
> If I flip over to a Kerberised environment, the NameNode and DataNodes,
> running as the 'hdfs' user, and the Resource and and Node Managers, running
> as the 'yarn' user, all start up OK and their respective web exposure can
> be
> used.
>
>
> When I try to start up the JobHistory server however
>
> /bin/su mapred -c
> '/local/Hadoop/hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh --config
> /local/Hadoop/hadoop-2.8.0/etc/hadoop/ start historyserver
>
> I get a message in the logs telling me that, rather than the mapred
> user doing things,
> a user 'jhs' is trying to do stuff, vis
>
> 2017-07-20 18:15:09,667 INFO
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX
> signal handlers for [TERM, HUP, INT]
> 2017-07-20 18:15:10,062 INFO
> org.apache.hadoop.security.UserGroupInformation: Login successful for
> user jhs/co246a-9.ecs.vuw.ac.nz@ECS.VUW.AC.NZ using keytab file
> /local/Hadoop/krb/jhs.service.keytab
> 2017-07-20 18:15:10,107 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2017-07-20 18:15:10,142 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric
> snapshot period at 10 second(s).
> 2017-07-20 18:15:10,142 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer
> metrics system started
> 2017-07-20 18:15:10,145 INFO
> org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
> 2017-07-20 18:15:10,411 INFO
> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default
> file system [hdfs://co246a-a.ecs.vuw.ac.nz:9000]
> 2017-07-20 18:15:10,518 INFO
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state
> INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> Error creating done directory:
> [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating
> done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done]
> at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> tryCreatingHistoryDirs(HistoryFileManager.java:639)
> at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> createHistoryDirs(HistoryFileManager.java:585)
> at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.
> serviceInit(HistoryFileManager.java:550)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(
> JobHistory.java:95)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.hadoop.service.CompositeService.serviceInit(
> CompositeService.java:107)
> at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.
> serviceInit(JobHistoryServer.java:151)
> at org.apache.hadoop.service.AbstractService.init(
> AbstractService.java:163)
> at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.
> launchJobHistoryServer(JobHistoryServer.java:231)
> at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(
> JobHistoryServer.java:241)
> Caused by: org.apache.hadoop.security.AccessControlException:
> Permission denied: user=jhs, access=EXECUTE,
> inode="/mr-history":mapred:hadoop:drwxrwx---
>
>
> But where has the jhs user come from ?
>
> Doesn't appear to be set anywhere in any of the config files.
>
> According to the hadoop-2.8.0 docs SecureMode page,
>
> https://hadoop.apache.org/docs/r2.8.0/hadoop-project-
> dist/hadoop-common/SecureMode.html
>
> =============================================
> MapReduce JobHistory Server
>
> The MapReduce JobHistory Server keytab file, on that host, should look
> like the following:
>
> $ klist -e -k -t /etc/security/keytab/jhs.service.keytab
> Keytab name: FILE:/etc/security/keytab/jhs.service.keytab
> KVNO Timestamp Principal
> 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (AES-256 CTS mode with 96-bit SHA-1 HMAC)
> 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (AES-128 CTS mode with 96-bit SHA-1 HMAC)
> 4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD
> (ArcFour with HMAC/md5)
> 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (AES-256 CTS mode with 96-bit SHA-1 HMAC)
> 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (AES-128 CTS mode with 96-bit SHA-1 HMAC)
> 4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD
> (ArcFour with HMAC/md5)
> =============================================
>
>
> and mine does.
>
> The hadoop-2.8.0 docs SecureMode page also suggests that one would need to
> play around with the
>
> hadoop.security.auth_to_local
>
> config value, but I haven't had to do that for the nn, dn, rm or nm
> keytabs.
>
> So is there something special about the jhs user ?
>
> Or perhaps something special about the other keytab values ?
>
> Any clues/insight welcome,
> Kevin
>
> ---
> Kevin M. Buckley
>
> eScience Consultant
> School of Engineering and Computer Science
> Victoria University of Wellington
> New Zealand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>