You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Matteo Durighetto <m....@miriade.it> on 2017/10/11 20:24:32 UTC
kudu 1.4 kerberos
Hello,
I have a strange behaviour with Kudu 1.4 and kerberos.
I enabled kerberos on kudu, I have the principal correctly in the OU of an
AD, but
at startup I got a lot of errors on method TSHeartbeat between tablet
server and
master server as unauthorized. There's no firewall between nodes.
W1011 <time> server_base.cc:316] Unauthorized access attempt
to method kudu.master.MasterService.TSHeartbeat
from {username='abcdefgh1234', principal='kudu/HOSTNAME@DOMAIN.XYZ'}
at <IP>:37360
the "abcdefgh1234" it's an example of the the string created by the
cloudera manager during the enable kerberos.
The other services (hdfs and so on ) are under kerberos without problem and
there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ).
As I understand the problem is something about the 3° level of
authorization between master servers and tablet servers.
So I try to understand better how it works.
The server_base.cc:316 is this call on branch 1.4 :
void ServerBase::LogUnauthorizedAccess(rpc::RpcContext* rpc) const {
LOG(WARNING) << "Unauthorized access attempt to method "
<< rpc->service_name() << "." << rpc->method_name()
<< " from " << rpc->requestor_string();
}
was called by :
bool ServerBase::Authorize(rpc::RpcContext* rpc, uint32_t allowed_roles) {
if ((allowed_roles & SUPER_USER) &&
superuser_acl_.UserAllowed(rpc->remote_user().username())) {
return true;
}
if ((allowed_roles & USER) &&
user_acl_.UserAllowed(rpc->remote_user().username())) {
return true;
}
if ((allowed_roles & SERVICE_USER) &&
service_acl_.UserAllowed(rpc->remote_user().username())) {
return true;
}
LogUnauthorizedAccess(rpc);
rpc->RespondFailure(Status::NotAuthorized("unauthorized access to method",
rpc->method_name()));
return false;
}
The part of the code called for a service is this :
if ((allowed_roles & SERVICE_USER) &&
service_acl_.UserAllowed(rpc->remote_user().username())) {
return true;
}
the service_acl_.UserAllowed(rpc->remote_user().username()) I think in
the could be
the problem.
because :
It is a kerberized service so, with the keytab ( for exemple with kudu/
HOSTNAME@DOMAIN.XYZ )
rpc->remote_user().username()
it will run this code ( rpc/remote_user.cc )
string RemoteUser::ToString() const {
string ret;
strings::SubstituteAndAppend(&ret, "{username='$0'", username_);
if (principal_) {
strings::SubstituteAndAppend(&ret, ", principal='$0'", *principal_);
}
ret.append("}");
return ret;
}
and it will returns the "{username='abcdefgh1234',principal='kudu/HOSTNAME@
DOMAIN.XYZ'} "
as in the error message
and the code service_acl_.UserAllowed call (src/kudu/security/simple_acl.cc
)
bool SimpleAcl::UserAllowed(const string& username) {
return ContainsKey(users_, "*") || ContainsKey(users_, username);
}
==> users_ , I not understand for service where is builded or read
the other part of code :
the allowed_roles & SERVICE_USER is a bitwise operation,
As I understand from server/server_base.h
SERVICE_USER = 1 << 2
kudu is using a bitmap: 100 for service user , 010 for user, and 001 for
superuser
enum {
SUPER_USER = 1,
USER = 1 << 1,
SERVICE_USER = 1 << 2
};
So I expectes it will be true.
So I think the problem, as I say before, could be in ContainsKey(users_,
username); :
bool SimpleAcl::UserAllowed(const string& username) {
return ContainsKey(users_, "*") || ContainsKey(users_, username);
}
At this point It's not clear for me how Kudu build the array/key list users
for daemon service ( it's not as super users or user ACL an external
parameter).
I would like to debug more, but I not have found more instructions than the
log level ( INFO ).
Kind Regards
Matteo
Re: kudu 1.4 kerberos
Posted by Matteo Durighetto <m....@miriade.it>.
Hello Todd,
after the workaround, we gone in production, I will
test asap in a new environment of test in the next days.
Kind regards
Matteo Durighetto
2017-10-24 21:46 GMT+02:00 Todd Lipcon <to...@cloudera.com>:
> On Tue, Oct 24, 2017 at 12:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> I've filed https://issues.apache.org/jira/browse/KUDU-2198 to provide a
>> workaround for systems like this. I should have a patch up shortly since
>> it's relatively simple.
>>
>>
> ... and here's the patch, if you want to try it out, Matteo:
> https://gerrit.cloudera.org/c/8373/
>
> -Todd
>
>
>> -Todd
>>
>> On Tue, Oct 17, 2017 at 7:00 PM, Brock Noland <br...@phdata.io> wrote:
>>
>>> Just one clarification below...
>>>
>>> > On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <
>>> m.durighetto@miriade.it> wrote:
>>> > the "abcdefgh1234" it's an example of the the string created by the
>>> cloudera manager during the enable kerberos.
>>>
>>> ...
>>>
>>> On Mon, Oct 16, 2017 at 11:57 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>> > Interesting. What is the sAMAccountName in this case? Wouldn't all of
>>> the 'kudu' have the same account name?
>>>
>>> CM generates some random names for cn and sAMAccountName. Below is an
>>> example created by CM.
>>>
>>> dn: CN=uQAtUOSwrA,OU=valhalla-kerberos,OU=Hadoop,DC=phdata,DC=io
>>> cn: uQAtUOSwrA
>>> sAMAccountName: uQAtUOSwrA
>>> userPrincipalName: kudu/worker5.valhalla.phdata.io@PHDATA.IO
>>> servicePrincipalName: kudu/worker5.valhalla.phdata.io
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: kudu 1.4 kerberos
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Oct 24, 2017 at 12:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
> I've filed https://issues.apache.org/jira/browse/KUDU-2198 to provide a
> workaround for systems like this. I should have a patch up shortly since
> it's relatively simple.
>
>
... and here's the patch, if you want to try it out, Matteo:
https://gerrit.cloudera.org/c/8373/
-Todd
> -Todd
>
> On Tue, Oct 17, 2017 at 7:00 PM, Brock Noland <br...@phdata.io> wrote:
>
>> Just one clarification below...
>>
>> > On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <
>> m.durighetto@miriade.it> wrote:
>> > the "abcdefgh1234" it's an example of the the string created by the
>> cloudera manager during the enable kerberos.
>>
>> ...
>>
>> On Mon, Oct 16, 2017 at 11:57 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> > Interesting. What is the sAMAccountName in this case? Wouldn't all of
>> the 'kudu' have the same account name?
>>
>> CM generates some random names for cn and sAMAccountName. Below is an
>> example created by CM.
>>
>> dn: CN=uQAtUOSwrA,OU=valhalla-kerberos,OU=Hadoop,DC=phdata,DC=io
>> cn: uQAtUOSwrA
>> sAMAccountName: uQAtUOSwrA
>> userPrincipalName: kudu/worker5.valhalla.phdata.io@PHDATA.IO
>> servicePrincipalName: kudu/worker5.valhalla.phdata.io
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: kudu 1.4 kerberos
Posted by Todd Lipcon <to...@cloudera.com>.
I've filed https://issues.apache.org/jira/browse/KUDU-2198 to provide a
workaround for systems like this. I should have a patch up shortly since
it's relatively simple.
-Todd
On Tue, Oct 17, 2017 at 7:00 PM, Brock Noland <br...@phdata.io> wrote:
> Just one clarification below...
>
> > On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <
> m.durighetto@miriade.it> wrote:
> > the "abcdefgh1234" it's an example of the the string created by the
> cloudera manager during the enable kerberos.
>
> ...
>
> On Mon, Oct 16, 2017 at 11:57 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > Interesting. What is the sAMAccountName in this case? Wouldn't all of
> the 'kudu' have the same account name?
>
> CM generates some random names for cn and sAMAccountName. Below is an
> example created by CM.
>
> dn: CN=uQAtUOSwrA,OU=valhalla-kerberos,OU=Hadoop,DC=phdata,DC=io
> cn: uQAtUOSwrA
> sAMAccountName: uQAtUOSwrA
> userPrincipalName: kudu/worker5.valhalla.phdata.io@PHDATA.IO
> servicePrincipalName: kudu/worker5.valhalla.phdata.io
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: kudu 1.4 kerberos
Posted by Brock Noland <br...@phdata.io>.
Just one clarification below...
> On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <m....@miriade.it> wrote:
> the "abcdefgh1234" it's an example of the the string created by the cloudera manager during the enable kerberos.
...
On Mon, Oct 16, 2017 at 11:57 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Interesting. What is the sAMAccountName in this case? Wouldn't all of the 'kudu' have the same account name?
CM generates some random names for cn and sAMAccountName. Below is an
example created by CM.
dn: CN=uQAtUOSwrA,OU=valhalla-kerberos,OU=Hadoop,DC=phdata,DC=io
cn: uQAtUOSwrA
sAMAccountName: uQAtUOSwrA
userPrincipalName: kudu/worker5.valhalla.phdata.io@PHDATA.IO
servicePrincipalName: kudu/worker5.valhalla.phdata.io
Re: kudu 1.4 kerberos
Posted by Matteo Durighetto <m....@miriade.it>.
Hello Todd,
because our systems are join in AD Domain with red hat
SSSD, this system map the principal bypassing krb5.conf
and map every principal to an univocal sAMAccount, the sAMAccount is a
unique attribute of the entry of an AD record ( https://blogs.msdn.microsoft
.com/openspecification/2009/07/10/understanding-unique-att
ributes-in-active-directory/ ), so its not possible to have the same
sAMAccount for different kudu principals
By default SSSD introduce in krb5.conf this entry :
includedir /var/lib/sss/pubconf/krb5.include.d/
And with this entry it include the :
cat /var/lib/sss/pubconf/krb5.include.d/localauth_plugin
[plugins]
localauth = {
module = sssd:/usr/lib64/sssd/modules/sssd_krb5_localauth_plugin.so
}
So the auth_to_local krb5 api return the sAMAccount, instead of "kudu" for
a principal like kudu/<hostname>@domain
I created a new krb5.conf withtout "includedir
/var/lib/sss/pubconf/krb5.include.d/"
and used that variable in KUDU env advanced setting and now it's work.
So for a new enviroment there a set of possible solutions :
1) create a new /etc/krb5_onlykudu.conf and set KRB5_CONFIG=/etc/krb5_
onlykudu.conf
2) modify /etc/krb5.conf and remove the entry "includedir
/var/lib/sss/pubconf/krb5.include.d/
3) if there is the entry "includedir /var/lib/sss/pubconf/krb5.include.d/"
in /etc/krb5.conf , remove the file "/var/lib/sss/pubconf/krb5.
include.d/localauth_plugin"
Thank you very much for your support
Kind Regards
Matteo Durighetto
"Logged in from keytab as kudu/<host>@REALM (short username
>> <sAMAccountName> )"
>>
>> I begin to think that the problem is between sssd with plugin
>> sssd_krb5_localauth_plugin.so
>> so for every principal kudu/<host>@REALM kudu maps to <sAMAccountName>,
>> so seems impossible to have the all kudu/<host>@REALM mapped to the same
>> "kudu"
>> as suggested "So, basically, it's critical that the username that the
>> master determines for itself (from this function)
>> matches the username that it has determined for the tablet servers when
>> they authenticate
>> (what you pasted as 'abcdefgh1234' above)."
>>
>
> Interesting. What is the sAMAccountName in this case? Wouldn't all of the
> 'kudu' have the same account name?
>
>
>>
>> The strange thing is that with hadoop I have the correct mapping (
>> probably because I have no rule, so
>> it switch to default rule )
>>
>> hadoop org.apache.hadoop.security.HadoopKerberosName kudu/<host>@REALM
>> ==> kudu
>>
>>
> This makes sense since Hadoop uses its own auth_to_local configuration
> defined in its core-site.xml rather than relying on the system krb5
> library. Kudu, being implemented in C++, uses the system krb5 API to do
> mapping, and hence picks upthe sssd localauth plugin as you found.
>
> As a temporary workaround, one thing you could try is to duplicate your
> krb5.conf and make a new one like krb5.kudu.conf. In this file remove the
> localauth plugin and any auth_to_local mappings that are causing trouble.
> You can then start the kudu servers with KRB5_CONFIG=/path/to/krb5.kudu.conf
> . I believe this will ensure that the local-auth mapping performs as you'd
> like.
>
> Given your setup, do you think we should provide an advanced option to
> override the system-configured localauth mapping and instead just use the
> "simple" mapping of using the short principal name? Generally we'd prefer
> to have as simple a configuration as possible but if your configuration is
> relatively commonplace it seems we might want an easier workaround than
> duplicating krb5.conf.
>
> -Todd
>
>>
>>
>> 2017-10-13 1:32 GMT+02:00 Todd Lipcon <to...@cloudera.com>:
>>
>>> Hey Matteo,
>>>
>>> Looks like you did quite a bit of digging in the code! Responses inline
>>> below.
>>>
>>> On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <
>>> m.durighetto@miriade.it> wrote:
>>>
>>>> Hello,
>>>> I have a strange behaviour with Kudu 1.4 and kerberos.
>>>> I enabled kerberos on kudu, I have the principal correctly in the OU of
>>>> an AD, but
>>>> at startup I got a lot of errors on method TSHeartbeat between tablet
>>>> server and
>>>> master server as unauthorized. There's no firewall between nodes.
>>>>
>>>
>>> right, "unauthorized" indicates that the connection was made fine, but
>>> the individual RPC call was determined to not be allowed for the identity
>>> presented on the other side of the connection.
>>>
>>>
>>>>
>>>> W1011 <time> server_base.cc:316] Unauthorized access attempt
>>>> to method kudu.master.MasterService.TSHeartbeat
>>>> from {username='abcdefgh1234', principal='kudu/HOSTNAME@DOMAIN.XYZ'}
>>>> at <IP>:37360
>>>>
>>>> the "abcdefgh1234" it's an example of the the string created by the
>>>> cloudera manager during the enable kerberos.
>>>>
>>>
>>> This output indicates that it successfully authenticated via Kerberos as
>>> the principal listed above. That's good news and means you don't need to
>>> worry about rdns, etc (if you had issues with that it would have had
>>> trouble finding a service ticket or authenticating the connection). This
>>> means you got past the "authentication" step and having problems at the
>>> "authorization" step.
>>>
>>>
>>>>
>>>> The other services (hdfs and so on ) are under kerberos without problem
>>>> and there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ).
>>>> As I understand the problem is something about the 3° level of
>>>> authorization between master servers and tablet servers.
>>>>
>>>
>>> Right.
>>>
>>>
>>>> ... <snipped>
>>>> So I think the problem, as I say before, could be
>>>> in ContainsKey(users_, username); :
>>>>
>>>> bool SimpleAcl::UserAllowed(const string& username) {
>>>> return ContainsKey(users_, "*") || ContainsKey(users_, username);
>>>> }
>>>>
>>>>
>>>> At this point It's not clear for me how Kudu build the array/key list
>>>> users for daemon service ( it's not as super users or user ACL an external
>>>> parameter).
>>>>
>>>
>>> Exactly. The users here for the 'service' ACL are set in
>>> ServerBase::InitAcls():
>>>
>>> boost::optional<string> keytab_user = security::GetLoggedInUsernameF
>>> romKeytab();
>>> if (keytab_user) {
>>> // If we're logged in from a keytab, then everyone should be, and we
>>> expect them
>>> // to use the same mapped username.
>>> service_user = *keytab_user;
>>> } else {
>>> // If we aren't logged in from a keytab, then just assume that the
>>> services
>>> // will be running as the same Unix user as we are.
>>> RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user),
>>> "could not deterine local username");
>>> }
>>>
>>> Since you're using Kerberos, the top branch here would apply -- it's
>>> calling GetLoggedInUsernameFromKeytab() from init.cc.
>>>
>>> You can see what username the server is getting by looking for a log
>>> message at startup like "Logged in from keytab as kudu/<host>@REALM (short
>>> username <XYZ>)". Here 'XYZ' is the username that ends up in the service
>>> ACL.
>>>
>>> So, basically, it's critical that the username that the master
>>> determines for itself (from this function) matches the username that it has
>>> determined for the tablet servers when they authenticate (what you pasted
>>> as 'abcdefgh1234' above).
>>>
>>> That brings us to the next question: how do we convert from a principal
>>> like kudu/<HOST>@REALM to a short "username"? The answer there is the
>>> function 'MapPrincipalToLocalName' again from security/init.cc. This
>>> function delegates the mapping to the krb5 library itself using the
>>> krb5_aname_to_localname() API. The results of this API can vary depending
>>> on the kerberos configuration, but in typical configurations it's
>>> determined by the 'auth_to_local' configuration in your krb5.conf. See the
>>> corresponding section in the docs here:
>>>
>>> https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/
>>> krb5_conf.html
>>>
>>> My guess is that your host has been configured such that when the master
>>> maps its own principal, it's getting a different result than when it maps
>>> the principal being used by the tservers.
>>>
>>> Hope that gets you on the right track.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: kudu 1.4 kerberos
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Oct 16, 2017 at 2:29 PM, Matteo Durighetto <m....@miriade.it>
wrote:
> Hello Todd,
> thank you very much for the answer. I think I have
> found something interesting.
>
> Kudu is doing the ACL list with the sAMAccountName or CN as it writes in
> the logs:
>
> "Logged in from keytab as kudu/<host>@REALM (short username
> <sAMAccountName> )"
>
> I begin to think that the problem is between sssd with plugin
> sssd_krb5_localauth_plugin.so
> so for every principal kudu/<host>@REALM kudu maps to <sAMAccountName>,
> so seems impossible to have the all kudu/<host>@REALM mapped to the same
> "kudu"
> as suggested "So, basically, it's critical that the username that the
> master determines for itself (from this function)
> matches the username that it has determined for the tablet servers when
> they authenticate
> (what you pasted as 'abcdefgh1234' above)."
>
Interesting. What is the sAMAccountName in this case? Wouldn't all of the
'kudu' have the same account name?
>
> The strange thing is that with hadoop I have the correct mapping (
> probably because I have no rule, so
> it switch to default rule )
>
> hadoop org.apache.hadoop.security.HadoopKerberosName kudu/<host>@REALM
> ==> kudu
>
>
This makes sense since Hadoop uses its own auth_to_local configuration
defined in its core-site.xml rather than relying on the system krb5
library. Kudu, being implemented in C++, uses the system krb5 API to do
mapping, and hence picks upthe sssd localauth plugin as you found.
As a temporary workaround, one thing you could try is to duplicate your
krb5.conf and make a new one like krb5.kudu.conf. In this file remove the
localauth plugin and any auth_to_local mappings that are causing trouble.
You can then start the kudu servers with
KRB5_CONFIG=/path/to/krb5.kudu.conf . I believe this will ensure that the
local-auth mapping performs as you'd like.
Given your setup, do you think we should provide an advanced option to
override the system-configured localauth mapping and instead just use the
"simple" mapping of using the short principal name? Generally we'd prefer
to have as simple a configuration as possible but if your configuration is
relatively commonplace it seems we might want an easier workaround than
duplicating krb5.conf.
-Todd
>
>
> 2017-10-13 1:32 GMT+02:00 Todd Lipcon <to...@cloudera.com>:
>
>> Hey Matteo,
>>
>> Looks like you did quite a bit of digging in the code! Responses inline
>> below.
>>
>> On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <
>> m.durighetto@miriade.it> wrote:
>>
>>> Hello,
>>> I have a strange behaviour with Kudu 1.4 and kerberos.
>>> I enabled kerberos on kudu, I have the principal correctly in the OU of
>>> an AD, but
>>> at startup I got a lot of errors on method TSHeartbeat between tablet
>>> server and
>>> master server as unauthorized. There's no firewall between nodes.
>>>
>>
>> right, "unauthorized" indicates that the connection was made fine, but
>> the individual RPC call was determined to not be allowed for the identity
>> presented on the other side of the connection.
>>
>>
>>>
>>> W1011 <time> server_base.cc:316] Unauthorized access attempt
>>> to method kudu.master.MasterService.TSHeartbeat
>>> from {username='abcdefgh1234', principal='kudu/HOSTNAME@DOMAIN.XYZ'}
>>> at <IP>:37360
>>>
>>> the "abcdefgh1234" it's an example of the the string created by the
>>> cloudera manager during the enable kerberos.
>>>
>>
>> This output indicates that it successfully authenticated via Kerberos as
>> the principal listed above. That's good news and means you don't need to
>> worry about rdns, etc (if you had issues with that it would have had
>> trouble finding a service ticket or authenticating the connection). This
>> means you got past the "authentication" step and having problems at the
>> "authorization" step.
>>
>>
>>>
>>> The other services (hdfs and so on ) are under kerberos without problem
>>> and there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ).
>>> As I understand the problem is something about the 3° level of
>>> authorization between master servers and tablet servers.
>>>
>>
>> Right.
>>
>>
>>> ... <snipped>
>>> So I think the problem, as I say before, could be
>>> in ContainsKey(users_, username); :
>>>
>>> bool SimpleAcl::UserAllowed(const string& username) {
>>> return ContainsKey(users_, "*") || ContainsKey(users_, username);
>>> }
>>>
>>>
>>> At this point It's not clear for me how Kudu build the array/key list
>>> users for daemon service ( it's not as super users or user ACL an external
>>> parameter).
>>>
>>
>> Exactly. The users here for the 'service' ACL are set in
>> ServerBase::InitAcls():
>>
>> boost::optional<string> keytab_user = security::GetLoggedInUsernameF
>> romKeytab();
>> if (keytab_user) {
>> // If we're logged in from a keytab, then everyone should be, and we
>> expect them
>> // to use the same mapped username.
>> service_user = *keytab_user;
>> } else {
>> // If we aren't logged in from a keytab, then just assume that the
>> services
>> // will be running as the same Unix user as we are.
>> RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user),
>> "could not deterine local username");
>> }
>>
>> Since you're using Kerberos, the top branch here would apply -- it's
>> calling GetLoggedInUsernameFromKeytab() from init.cc.
>>
>> You can see what username the server is getting by looking for a log
>> message at startup like "Logged in from keytab as kudu/<host>@REALM (short
>> username <XYZ>)". Here 'XYZ' is the username that ends up in the service
>> ACL.
>>
>> So, basically, it's critical that the username that the master determines
>> for itself (from this function) matches the username that it has determined
>> for the tablet servers when they authenticate (what you pasted as
>> 'abcdefgh1234' above).
>>
>> That brings us to the next question: how do we convert from a principal
>> like kudu/<HOST>@REALM to a short "username"? The answer there is the
>> function 'MapPrincipalToLocalName' again from security/init.cc. This
>> function delegates the mapping to the krb5 library itself using the
>> krb5_aname_to_localname() API. The results of this API can vary depending
>> on the kerberos configuration, but in typical configurations it's
>> determined by the 'auth_to_local' configuration in your krb5.conf. See the
>> corresponding section in the docs here:
>>
>> https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/
>> krb5_conf.html
>>
>> My guess is that your host has been configured such that when the master
>> maps its own principal, it's getting a different result than when it maps
>> the principal being used by the tservers.
>>
>> Hope that gets you on the right track.
>>
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: kudu 1.4 kerberos
Posted by Matteo Durighetto <m....@miriade.it>.
Hello Todd,
thank you very much for the answer. I think I have found
something interesting.
Kudu is doing the ACL list with the sAMAccountName or CN as it writes in
the logs:
"Logged in from keytab as kudu/<host>@REALM (short username
<sAMAccountName> )"
I begin to think that the problem is between sssd with plugin
sssd_krb5_localauth_plugin.so
so for every principal kudu/<host>@REALM kudu maps to <sAMAccountName>,
so seems impossible to have the all kudu/<host>@REALM mapped to the same
"kudu"
as suggested "So, basically, it's critical that the username that the
master determines for itself (from this function)
matches the username that it has determined for the tablet servers when
they authenticate
(what you pasted as 'abcdefgh1234' above)."
The strange thing is that with hadoop I have the correct mapping ( probably
because I have no rule, so
it switch to default rule )
hadoop org.apache.hadoop.security.HadoopKerberosName kudu/<host>@REALM
==> kudu
Matteo Durighetto
2017-10-13 1:32 GMT+02:00 Todd Lipcon <to...@cloudera.com>:
> Hey Matteo,
>
> Looks like you did quite a bit of digging in the code! Responses inline
> below.
>
> On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <
> m.durighetto@miriade.it> wrote:
>
>> Hello,
>> I have a strange behaviour with Kudu 1.4 and kerberos.
>> I enabled kerberos on kudu, I have the principal correctly in the OU of
>> an AD, but
>> at startup I got a lot of errors on method TSHeartbeat between tablet
>> server and
>> master server as unauthorized. There's no firewall between nodes.
>>
>
> right, "unauthorized" indicates that the connection was made fine, but the
> individual RPC call was determined to not be allowed for the identity
> presented on the other side of the connection.
>
>
>>
>> W1011 <time> server_base.cc:316] Unauthorized access attempt
>> to method kudu.master.MasterService.TSHeartbeat
>> from {username='abcdefgh1234', principal='kudu/HOSTNAME@DOMAIN.XYZ'}
>> at <IP>:37360
>>
>> the "abcdefgh1234" it's an example of the the string created by the
>> cloudera manager during the enable kerberos.
>>
>
> This output indicates that it successfully authenticated via Kerberos as
> the principal listed above. That's good news and means you don't need to
> worry about rdns, etc (if you had issues with that it would have had
> trouble finding a service ticket or authenticating the connection). This
> means you got past the "authentication" step and having problems at the
> "authorization" step.
>
>
>>
>> The other services (hdfs and so on ) are under kerberos without problem
>> and there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ).
>> As I understand the problem is something about the 3° level of
>> authorization between master servers and tablet servers.
>>
>
> Right.
>
>
>> ... <snipped>
>> So I think the problem, as I say before, could be in ContainsKey(users_,
>> username); :
>>
>> bool SimpleAcl::UserAllowed(const string& username) {
>> return ContainsKey(users_, "*") || ContainsKey(users_, username);
>> }
>>
>>
>> At this point It's not clear for me how Kudu build the array/key list
>> users for daemon service ( it's not as super users or user ACL an external
>> parameter).
>>
>
> Exactly. The users here for the 'service' ACL are set in
> ServerBase::InitAcls():
>
> boost::optional<string> keytab_user = security::
> GetLoggedInUsernameFromKeytab();
> if (keytab_user) {
> // If we're logged in from a keytab, then everyone should be, and we
> expect them
> // to use the same mapped username.
> service_user = *keytab_user;
> } else {
> // If we aren't logged in from a keytab, then just assume that the
> services
> // will be running as the same Unix user as we are.
> RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user),
> "could not deterine local username");
> }
>
> Since you're using Kerberos, the top branch here would apply -- it's
> calling GetLoggedInUsernameFromKeytab() from init.cc.
>
> You can see what username the server is getting by looking for a log
> message at startup like "Logged in from keytab as kudu/<host>@REALM (short
> username <XYZ>)". Here 'XYZ' is the username that ends up in the service
> ACL.
>
> So, basically, it's critical that the username that the master determines
> for itself (from this function) matches the username that it has determined
> for the tablet servers when they authenticate (what you pasted as
> 'abcdefgh1234' above).
>
> That brings us to the next question: how do we convert from a principal
> like kudu/<HOST>@REALM to a short "username"? The answer there is the
> function 'MapPrincipalToLocalName' again from security/init.cc. This
> function delegates the mapping to the krb5 library itself using the
> krb5_aname_to_localname() API. The results of this API can vary depending
> on the kerberos configuration, but in typical configurations it's
> determined by the 'auth_to_local' configuration in your krb5.conf. See the
> corresponding section in the docs here:
>
> https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html
>
> My guess is that your host has been configured such that when the master
> maps its own principal, it's getting a different result than when it maps
> the principal being used by the tservers.
>
> Hope that gets you on the right track.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: kudu 1.4 kerberos
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Matteo,
Looks like you did quite a bit of digging in the code! Responses inline
below.
On Wed, Oct 11, 2017 at 1:24 PM, Matteo Durighetto <m....@miriade.it>
wrote:
> Hello,
> I have a strange behaviour with Kudu 1.4 and kerberos.
> I enabled kerberos on kudu, I have the principal correctly in the OU of an
> AD, but
> at startup I got a lot of errors on method TSHeartbeat between tablet
> server and
> master server as unauthorized. There's no firewall between nodes.
>
right, "unauthorized" indicates that the connection was made fine, but the
individual RPC call was determined to not be allowed for the identity
presented on the other side of the connection.
>
> W1011 <time> server_base.cc:316] Unauthorized access attempt
> to method kudu.master.MasterService.TSHeartbeat
> from {username='abcdefgh1234', principal='kudu/HOSTNAME@DOMAIN.XYZ'}
> at <IP>:37360
>
> the "abcdefgh1234" it's an example of the the string created by the
> cloudera manager during the enable kerberos.
>
This output indicates that it successfully authenticated via Kerberos as
the principal listed above. That's good news and means you don't need to
worry about rdns, etc (if you had issues with that it would have had
trouble finding a service ticket or authenticating the connection). This
means you got past the "authentication" step and having problems at the
"authorization" step.
>
> The other services (hdfs and so on ) are under kerberos without problem
> and there is the rdns at true in the /etc/krb5.conf ( KUDU-2032 ).
> As I understand the problem is something about the 3° level of
> authorization between master servers and tablet servers.
>
Right.
> ... <snipped>
> So I think the problem, as I say before, could be in ContainsKey(users_,
> username); :
>
> bool SimpleAcl::UserAllowed(const string& username) {
> return ContainsKey(users_, "*") || ContainsKey(users_, username);
> }
>
>
> At this point It's not clear for me how Kudu build the array/key list
> users for daemon service ( it's not as super users or user ACL an external
> parameter).
>
Exactly. The users here for the 'service' ACL are set in
ServerBase::InitAcls():
boost::optional<string> keytab_user =
security::GetLoggedInUsernameFromKeytab();
if (keytab_user) {
// If we're logged in from a keytab, then everyone should be, and we
expect them
// to use the same mapped username.
service_user = *keytab_user;
} else {
// If we aren't logged in from a keytab, then just assume that the
services
// will be running as the same Unix user as we are.
RETURN_NOT_OK_PREPEND(GetLoggedInUser(&service_user),
"could not deterine local username");
}
Since you're using Kerberos, the top branch here would apply -- it's
calling GetLoggedInUsernameFromKeytab() from init.cc.
You can see what username the server is getting by looking for a log
message at startup like "Logged in from keytab as kudu/<host>@REALM (short
username <XYZ>)". Here 'XYZ' is the username that ends up in the service
ACL.
So, basically, it's critical that the username that the master determines
for itself (from this function) matches the username that it has determined
for the tablet servers when they authenticate (what you pasted as
'abcdefgh1234' above).
That brings us to the next question: how do we convert from a principal
like kudu/<HOST>@REALM to a short "username"? The answer there is the
function 'MapPrincipalToLocalName' again from security/init.cc. This
function delegates the mapping to the krb5 library itself using the
krb5_aname_to_localname() API. The results of this API can vary depending
on the kerberos configuration, but in typical configurations it's
determined by the 'auth_to_local' configuration in your krb5.conf. See the
corresponding section in the docs here:
https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html
My guess is that your host has been configured such that when the master
maps its own principal, it's getting a different result than when it maps
the principal being used by the tservers.
Hope that gets you on the right track.
Thanks
-Todd
--
Todd Lipcon
Software Engineer, Cloudera