You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Steve Howard <st...@gmail.com> on 2015/05/27 16:55:21 UTC
Active Directory as a KDC for Hadoop
Hi All,
We are having an issue with the Ambari 2.0 release, and its wizard to
configure Active Directory as a KDC for securing the cluster. We had no
errors during configuration, but none of the services start after it has
been completed.
Specifically, we get the infamous "Client not found in Kerberos database"
message. This is actually a very simple one node cluster with Ambari and
HDP on Centos 6. We point to a Windows Server 2008 AD DC. When we print
the associated attributes in AD, it looks like the UPN is formatted as a
service principal name, which I don't think AD supports.
See below for a snippet of the attributes in AD...
[root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a |
grep nn
>>>"CN=nn/ambari2.howard.local,CN=Users"
cn: nn/ambari2.howard.local
userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL
servicePrincipalName: nn/ambari2.howard.local
distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
name: nn/ambari2.howard.local
[root@ambari2 ~]#
Has anyone run in this? Conversely, has anyone gotten AD to work as a KDC
for Hadoop?
Thanks,
Steve
Re: Active Directory as a KDC for Hadoop
Posted by Steve Howard <st...@gmail.com>.
I really wonder if this isn't related to AD 2008. Notice the attributes
(all of which are printed below) for the nn/_HOST/@REALM below. It has the
entry configured as a user schema, which sounds right for login. I am
going to test this against 2012, as perhaps that is the issue.
The only other idea I have is that this server is also joined to the AD
domain via winbind/samba, so perhaps that is related (although I don't see
why).
I would be interested to see if anyone else can successfully run
Hadoop/Kerberos against AD 2008.
-------------------------------
[root@ambari2 ~]# java TestAD | strings -a | awk '{if ($0 ~ "^>.*nn")
{f=1;print} else if (f == 1 && $0 !~ ">") {print} else if ($0 ~ ">" && f ==
1) {exit}}'
>>>"CN=nn/ambari2.howard.local,CN=Users"
sAMAccountType: 805306368
primaryGroupID: 513
objectClass: top, person, organizationalPerson, user
badPasswordTime: 130771268549472640
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=howard,DC=local
cn: nn/ambari2.howard.local
userAccountControl: 66048
userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL
servicePrincipalName: nn/ambari2.howard.local
dSCorePropagationData: 16010101000000.0Z
codePage: 0
distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
whenChanged: 20150526155101.0Z
whenCreated: 20150525122743.0Z
pwdLastSet: 130771264637265610
logonCount: 2
accountExpires: 0
lastLogoff: 0
lastLogonTimestamp: 130771290611601540
objectGUID: )
lastLogon: 130771290612539040
uSNChanged: 196192
uSNCreated: 194149
objectSid:
countryCode: 0
sAMAccountName: $G41000-F1M18MJHSNA6
instanceType: 4
badPwdCount: 0
name: nn/ambari2.howard.local
On Wed, May 27, 2015 at 1:31 PM, Steve Howard <st...@gmail.com>
wrote:
> Hi Bob,
>
> Thanks for the quick reply. My first thought was that it would be DNS
> related or something similar, but I can successfully connect/authenticate
> when I compiled a command line client class with a "normal"
> userPrincipalName account and an associated keytab. When I change the same
> test class to use the UPN generated by Ambari and its associated keytab, it
> always throws the exception listed.
>
> We also have a ticket open with HortonWorks support, but thought the list
> may be as quick in terms of a direction to pursue. I will reply back when
> we get more info.
>
> Thanks,
>
> Steve
>
> On Wed, May 27, 2015 at 1:20 PM, Robert Levas <rl...@hortonworks.com>
> wrote:
>
>> Hi Steve…
>>
>> We have successfully enable Kerberos on many clusters using AD as the
>> KDC. My experience is with Windows Server 2012, though.
>>
>> The details you are showing for the NN service identity looks correct,
>> so I don’t think that is an issue. If it wasn’t, Active Directory would
>> have rejected it upon creation of the account. However if you believe that
>> the UPN is incorrect, you can disable Kerberos and then re-enbable
>> Kerberos. However on the 2nd Wizard screen you should edit the "Attribute
>> template” under the "Advanced kerberos-env” section and change:
>>
>> *Original*: "userPrincipalName": "$normalized_principal",
>> *Updated*: "userPrincipalName": "$principal_name",
>>
>> The “Client not found in Kerberos database” indicates that the identity
>> in question may not have been created. There may be several reason for
>> this… maybe the UPN is incorrect, maybe the host cannot communicate with
>> the AD (this could happen if the krb5.conf file is incorrect).
>>
>> I hope this helps,
>> Rob
>>
>>
>> From: Steve Howard <st...@gmail.com>
>> Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
>> Date: Wednesday, May 27, 2015 at 10:55 AM
>> To: "user@ambari.apache.org" <us...@ambari.apache.org>
>> Subject: Active Directory as a KDC for Hadoop
>>
>> Hi All,
>>
>> We are having an issue with the Ambari 2.0 release, and its wizard to
>> configure Active Directory as a KDC for securing the cluster. We had no
>> errors during configuration, but none of the services start after it has
>> been completed.
>>
>> Specifically, we get the infamous "Client not found in Kerberos
>> database" message. This is actually a very simple one node cluster with
>> Ambari and HDP on Centos 6. We point to a Windows Server 2008 AD DC. When
>> we print the associated attributes in AD, it looks like the UPN is
>> formatted as a service principal name, which I don't think AD supports.
>>
>> See below for a snippet of the attributes in AD...
>>
>> [root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a |
>> grep nn
>> >>>"CN=nn/ambari2.howard.local,CN=Users"
>> cn: nn/ambari2.howard.local
>> userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL
>> servicePrincipalName: nn/ambari2.howard.local
>> distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
>>
>> name: nn/ambari2.howard.local
>> [root@ambari2 ~]#
>>
>> Has anyone run in this? Conversely, has anyone gotten AD to work as a
>> KDC for Hadoop?
>>
>> Thanks,
>>
>> Steve
>>
>
>
Re: Active Directory as a KDC for Hadoop
Posted by Steve Howard <st...@gmail.com>.
Hi Bob,
Thanks for the quick reply. My first thought was that it would be DNS
related or something similar, but I can successfully connect/authenticate
when I compiled a command line client class with a "normal"
userPrincipalName account and an associated keytab. When I change the same
test class to use the UPN generated by Ambari and its associated keytab, it
always throws the exception listed.
We also have a ticket open with HortonWorks support, but thought the list
may be as quick in terms of a direction to pursue. I will reply back when
we get more info.
Thanks,
Steve
On Wed, May 27, 2015 at 1:20 PM, Robert Levas <rl...@hortonworks.com>
wrote:
> Hi Steve…
>
> We have successfully enable Kerberos on many clusters using AD as the
> KDC. My experience is with Windows Server 2012, though.
>
> The details you are showing for the NN service identity looks correct,
> so I don’t think that is an issue. If it wasn’t, Active Directory would
> have rejected it upon creation of the account. However if you believe that
> the UPN is incorrect, you can disable Kerberos and then re-enbable
> Kerberos. However on the 2nd Wizard screen you should edit the "Attribute
> template” under the "Advanced kerberos-env” section and change:
>
> *Original*: "userPrincipalName": "$normalized_principal",
> *Updated*: "userPrincipalName": "$principal_name",
>
> The “Client not found in Kerberos database” indicates that the identity
> in question may not have been created. There may be several reason for
> this… maybe the UPN is incorrect, maybe the host cannot communicate with
> the AD (this could happen if the krb5.conf file is incorrect).
>
> I hope this helps,
> Rob
>
>
> From: Steve Howard <st...@gmail.com>
> Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
> Date: Wednesday, May 27, 2015 at 10:55 AM
> To: "user@ambari.apache.org" <us...@ambari.apache.org>
> Subject: Active Directory as a KDC for Hadoop
>
> Hi All,
>
> We are having an issue with the Ambari 2.0 release, and its wizard to
> configure Active Directory as a KDC for securing the cluster. We had no
> errors during configuration, but none of the services start after it has
> been completed.
>
> Specifically, we get the infamous "Client not found in Kerberos database"
> message. This is actually a very simple one node cluster with Ambari and
> HDP on Centos 6. We point to a Windows Server 2008 AD DC. When we print
> the associated attributes in AD, it looks like the UPN is formatted as a
> service principal name, which I don't think AD supports.
>
> See below for a snippet of the attributes in AD...
>
> [root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a |
> grep nn
> >>>"CN=nn/ambari2.howard.local,CN=Users"
> cn: nn/ambari2.howard.local
> userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL
> servicePrincipalName: nn/ambari2.howard.local
> distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
> name: nn/ambari2.howard.local
> [root@ambari2 ~]#
>
> Has anyone run in this? Conversely, has anyone gotten AD to work as a
> KDC for Hadoop?
>
> Thanks,
>
> Steve
>
Re: Active Directory as a KDC for Hadoop
Posted by Robert Levas <rl...@hortonworks.com>.
Hi Steve...
We have successfully enable Kerberos on many clusters using AD as the KDC. My experience is with Windows Server 2012, though.
The details you are showing for the NN service identity looks correct, so I don't think that is an issue. If it wasn't, Active Directory would have rejected it upon creation of the account. However if you believe that the UPN is incorrect, you can disable Kerberos and then re-enbable Kerberos. However on the 2nd Wizard screen you should edit the "Attribute template" under the "Advanced kerberos-env" section and change:
Original: "userPrincipalName": "$normalized_principal",
Updated: "userPrincipalName": "$principal_name",
The "Client not found in Kerberos database" indicates that the identity in question may not have been created. There may be several reason for this... maybe the UPN is incorrect, maybe the host cannot communicate with the AD (this could happen if the krb5.conf file is incorrect).
I hope this helps,
Rob
From: Steve Howard <st...@gmail.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Wednesday, May 27, 2015 at 10:55 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Active Directory as a KDC for Hadoop
Hi All,
We are having an issue with the Ambari 2.0 release, and its wizard to configure Active Directory as a KDC for securing the cluster. We had no errors during configuration, but none of the services start after it has been completed.
Specifically, we get the infamous "Client not found in Kerberos database" message. This is actually a very simple one node cluster with Ambari and HDP on Centos 6. We point to a Windows Server 2008 AD DC. When we print the associated attributes in AD, it looks like the UPN is formatted as a service principal name, which I don't think AD supports.
See below for a snippet of the attributes in AD...
[root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a | grep nn
>>>"CN=nn/ambari2.howard.local,CN=Users"
cn: nn/ambari2.howard.local
userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL<ma...@HOWARD.LOCAL>
servicePrincipalName: nn/ambari2.howard.local
distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
name: nn/ambari2.howard.local
[root@ambari2 ~]#
Has anyone run in this? Conversely, has anyone gotten AD to work as a KDC for Hadoop?
Thanks,
Steve
Re: Active Directory as a KDC for Hadoop
Posted by Robert Levas <rl...@hortonworks.com>.
Steve...
Thanks for the update on this.
Rob
From: Steve Howard <st...@gmail.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Thursday, May 28, 2015 at 9:12 PM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Re: Active Directory as a KDC for Hadoop
Just to close the loop on this, this is definitely an issue with how Server 2008 handles UPN's. As soon as I installed 2012 R2, with the exact same config, everything worked.
I had a ticket open with HortonWorks, and have asked them to add the Server 2012 requirement to the documentation for anyone that wants to secure a cluster with AD kerberos. Hopefully this will save someone else a lot of heartburn.
On Wed, May 27, 2015 at 10:55 AM, Steve Howard <st...@gmail.com>> wrote:
Hi All,
We are having an issue with the Ambari 2.0 release, and its wizard to configure Active Directory as a KDC for securing the cluster. We had no errors during configuration, but none of the services start after it has been completed.
Specifically, we get the infamous "Client not found in Kerberos database" message. This is actually a very simple one node cluster with Ambari and HDP on Centos 6. We point to a Windows Server 2008 AD DC. When we print the associated attributes in AD, it looks like the UPN is formatted as a service principal name, which I don't think AD supports.
See below for a snippet of the attributes in AD...
[root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a | grep nn
>>>"CN=nn/ambari2.howard.local,CN=Users"
cn: nn/ambari2.howard.local
userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL<ma...@HOWARD.LOCAL>
servicePrincipalName: nn/ambari2.howard.local
distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
name: nn/ambari2.howard.local
[root@ambari2 ~]#
Has anyone run in this? Conversely, has anyone gotten AD to work as a KDC for Hadoop?
Thanks,
Steve
Re: Active Directory as a KDC for Hadoop
Posted by Steve Howard <st...@gmail.com>.
Just to close the loop on this, this is definitely an issue with how Server
2008 handles UPN's. As soon as I installed 2012 R2, with the exact same
config, everything worked.
I had a ticket open with HortonWorks, and have asked them to add the Server
2012 requirement to the documentation for anyone that wants to secure a
cluster with AD kerberos. Hopefully this will save someone else a lot of
heartburn.
On Wed, May 27, 2015 at 10:55 AM, Steve Howard <st...@gmail.com>
wrote:
> Hi All,
>
> We are having an issue with the Ambari 2.0 release, and its wizard to
> configure Active Directory as a KDC for securing the cluster. We had no
> errors during configuration, but none of the services start after it has
> been completed.
>
> Specifically, we get the infamous "Client not found in Kerberos database"
> message. This is actually a very simple one node cluster with Ambari and
> HDP on Centos 6. We point to a Windows Server 2008 AD DC. When we print
> the associated attributes in AD, it looks like the UPN is formatted as a
> service principal name, which I don't think AD supports.
>
> See below for a snippet of the attributes in AD...
>
> [root@ambari2 ~]# /usr/jdk64/jdk1.7.0_67/bin/java TestAD | strings -a |
> grep nn
> >>>"CN=nn/ambari2.howard.local,CN=Users"
> cn: nn/ambari2.howard.local
> userPrincipalName: nn/ambari2.howard.local@HOWARD.LOCAL
> servicePrincipalName: nn/ambari2.howard.local
> distinguishedName: CN=nn/ambari2.howard.local,CN=Users,DC=howard,DC=local
> name: nn/ambari2.howard.local
> [root@ambari2 ~]#
>
> Has anyone run in this? Conversely, has anyone gotten AD to work as a KDC
> for Hadoop?
>
> Thanks,
>
> Steve
>