You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Philip Zeyliger <ph...@cloudera.com> on 2017/12/12 23:25:31 UTC
thrift-server-test
Hi folks,
I've been running into issues with thrift-server-test and Kerberos. Below
is an excerpt of "KRB5_TRACE=/dev/stderr
be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
SslConnectivity/2 fail the same way.
I'm running Ubuntu16.04. I've seen this both on my host, as well as inside
of an Ubuntu 16.04 Docker container.
Does this ring any bells?
Thanks!
-- Philip
[ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
Loading random data
Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
[31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
[31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): setting
up network...
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening
on fd 11: udp 0.0.0.0.51781 (pktinfo)
krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening
on fd 12: udp ::.51781 (pktinfo)
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set up 2
sockets
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
commencing operation
krb5kdc: starting...
Authenticating as principal philip/admin@KRBTEST.COM with password.
[31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
WARNING: no policy specified for impala/localhost@KRBTEST.COM; defaulting
to no policy
Principal "impala/localhost@KRBTEST.COM" created.
Authenticating as principal philip/admin@KRBTEST.COM with password.
[31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Entry for principal impala/localhost with kvno 2, encryption type
aes256-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
aes128-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
des3-cbc-sha1 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
arcfour-hmac added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ (6
etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922, etypes
{rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
KRBTEST.COM@KRBTEST.COM
[31476] 1513120922.532304: ccselect can't find appropriate cache for server
principal impala@localhost
[31476] 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
-> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
[31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
-1765328243/Matching credential not found
[31476] 1513120922.532407: Retrieving impala/localhost@KRBTEST.COM ->
krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found
[31476] 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM ->
krbtgt/KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
result: 0/Success
[31476] 1513120922.532441: Starting with TGT for client realm: impala/
localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM
[31476] 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found
[31476] 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM
using TGT krbtgt/KRBTEST.COM@KRBTEST.COM
[31476] 1513120922.532491: Generated subkey for TGS request: aes256-cts/005D
[31476] 1513120922.532524: etypes requested in TGS request: aes256-cts,
aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts
[31476] 1513120922.532574: Encoding request body and padata into FAST
request
[31476] 1513120922.532616: Sending request (951 bytes) to KRBTEST.COM
[31476] 1513120922.532630: Resolving hostname 127.0.0.1
[31476] 1513120922.532648: Sending initial UDP request to dgram
127.0.0.1:51781
[31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM ->
krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F
[31586] 1513120922.532814: Negotiated enctype based on authenticator:
aes256-cts
[31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): TGS_REQ
(6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server not
found in Kerberos database
[31476] 1513120922.533028: Received answer (491 bytes) from dgram
127.0.0.1:51781
[31476] 1513120922.533044: Response was not from master KDC
[31476] 1513120922.533053: Decoding FAST response
[31476] 1513120922.533081: TGS request result: -1765328377/Server krbtgt/
localhost@KRBTEST.COM not found in Kerberos database
/home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
Value of: status_.ok()
Actual: false
Expected: true
Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
more information (Server krbtgt/localhost@KRBTEST.COM not found in Kerberos
database))
[ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2,
where GetParam() = 2 (100 ms)
RE: thrift-server-test
Posted by Evo Eftimov <ev...@isecc.com>.
Ok but is your Impala daemon enrolled as Kerberos Principal in the KDC you are using for the tests and is the Impala Daemon also supplied with file with its Kerberos credentials
There are a number of error messages that Principal "impala/localhost@KRBTEST.COM can not be found
-----Original Message-----
From: Philip Zeyliger [mailto:philip@cloudera.com]
Sent: Wednesday, December 13, 2017 10:47 PM
To: dev@impala.apache.org
Subject: Re: thrift-server-test
The KDC in this case is the "minikdc" from https://github.com/apache/impala/blob/master/be/src/kudu/security/test/mini_kdc.cc.
I see evidence of it, and have been able to look at its configuration by, um, adding --gtest_break_on_failure. (The feature actually doesn't work, presumably because of an interaction with breakpad, but a temporary directory is left on my filesystem, so that's nice.)
-- Philip
On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <ev...@isecc.com> wrote:
> Is your cluster Kerberized at all, especially the Impala daemon - it
> doesn’t seem to be enrolled in the KDC at all
>
> You / your personal account/principal is definitely enrolled though
>
> And there is definetly a KDC in your environment
>
> -----Original Message-----
> From: Philip Zeyliger [mailto:philip@cloudera.com]
> Sent: Tuesday, December 12, 2017 11:26 PM
> To: dev@impala.apache.org
> Subject: thrift-server-test
>
> Hi folks,
>
> I've been running into issues with thrift-server-test and Kerberos.
> Below is an excerpt of "KRB5_TRACE=/dev/stderr
> be/build/debug/rpc/thrift-server-test";
> both SslConnectivity/1 and
> SslConnectivity/2 fail the same way.
>
> I'm running Ubuntu16.04. I've seen this both on my host, as well as
> inside of an Ubuntu 16.04 Docker container.
>
> Does this ring any bells?
>
> Thanks!
>
> -- Philip
>
>
> [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
> Loading random data
> Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
> KRBTEST.COM',
> master key name 'K/M@KRBTEST.COM'
> [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> result: 0/Success
> [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> result: 0/Success
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> setting up network...
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12:
> udp
> ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> philip-dev.gce.cloudera.com krb5kdc[31586](info):
> commencing operation
> krb5kdc: starting...
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> result: 0/Success
> WARNING: no policy specified for impala/localhost@KRBTEST.COM;
> defaulting to no policy Principal "impala/localhost@KRBTEST.COM" created.
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> result: 0/Success
> Entry for principal impala/localhost with kvno 2, encryption type
> aes256-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> aes128-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> des3-cbc-sha1 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> krb5kdc/impala_localhost.keytab.
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> AS_REQ
> (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> etypes
> {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't find
> appropriate cache for server principal impala@localhost [31476]
> 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
> -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> -1765328243/Matching credential not found [31476] 1513120922.532407:
> Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost
> from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found [31476]
> 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/
> KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> result: 0/Success
> [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476]
> 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found [31476]
> 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using
> TGT krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491:
> Generated subkey for TGS request: aes256-cts/005D [31476]
> 1513120922.532524: etypes requested in TGS request: aes256-cts,
> aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts
> [31476] 1513120922.532574: Encoding request body and padata into FAST request [31476] 1513120922.532616:
> Sending request (951 bytes) to KRBTEST.COM [31476] 1513120922.532630:
> Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending
> initial UDP request to dgram
> 127.0.0.1:51781
> [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM
> -> krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586]
> 1513120922.532814: Negotiated enctype based on authenticator:
> aes256-cts
> [31586] 1513120922.532820: Authenticator contains subkey:
> aes256-cts/005D Dec 12 15:22:02 philip-dev.gce.cloudera.com
> krb5kdc[31586](info): TGS_REQ
> (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server
> not found in Kerberos database [31476] 1513120922.533028: Received
> answer (491
> bytes) from dgram
> 127.0.0.1:51781
> [31476] 1513120922.533044: Response was not from master KDC [31476]
> 1513120922.533053: Decoding FAST response [31476] 1513120922.533081:
> TGS request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM
> not found in Kerberos database
> /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> failure: GSSAPI Error: Unspecified GSS failure. Minor code may
> provide more information (Server krbtgt/localhost@KRBTEST.COM not
> found in Kerberos
> database))
>
> [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2,
> where GetParam() = 2 (100 ms)
>
>
Re: thrift-server-test
Posted by Sailesh Mukil <sa...@cloudera.com>.
Your principal isn't getting cached for some reason. The same on my machine
looks like:
$ klist /tmp/krb5cc_impala_internal
Ticket cache: FILE:/tmp/krb5cc_impala_internal
Default principal: impala/localhost@KRBTEST.COM
Valid starting Expires Service principal
01/08/2018 15:49:30 01/09/2018 15:49:30 krbtgt/KRBTEST.COM@KRBTEST.COM
renew until 01/15/2018 15:49:30
01/08/2018 15:49:30 01/09/2018 15:49:30 impala/localhost@
renew until 01/15/2018 15:49:30
01/08/2018 15:49:30 01/09/2018 15:49:30 impala/localhost@KRBTEST.COM
renew until 01/15/2018 15:49:30
Did you check if your user has appropriate permissions on the file? Looks
like only the MiniKDC was able to write to it, and maybe your 'kinit'
couldn't and silently failed (which should be a bug if it didn't throw an
error) ?
On Mon, Jan 8, 2018 at 3:40 PM, Philip Zeyliger <ph...@cloudera.com> wrote:
> Hi Sailiesh,
>
> Is this what you'd expect?
>
> $klist /tmp/krb5cc_impala_internal
> Ticket cache: FILE:/tmp/krb5cc_impala_internal
> Default principal: impala/localhost@KRBTEST.COM
>
> Valid starting Expires Service principal
> 01/08/2018 15:39:23 01/09/2018 15:39:23 krbtgt/KRBTEST.COM@KRBTEST.COM
> renew until 01/15/2018 15:39:23
>
> Thanks!
>
>
> On Mon, Jan 8, 2018 at 12:20 PM, Sailesh Mukil <sa...@cloudera.com>
> wrote:
>
> > Can you run the test again, and klist the contents of the credential
> cache
> > and post the error logs again? Looks like "impala/localhost" might not be
> > stored as expected in the cache on your machine.
> >
> > On Wed, Dec 13, 2017 at 2:47 PM, Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> >
> > > The KDC in this case is the "minikdc" from
> > > https://github.com/apache/impala/blob/master/be/src/
> > > kudu/security/test/mini_kdc.cc.
> > > I see evidence of it, and have been able to look at its configuration
> by,
> > > um, adding --gtest_break_on_failure. (The feature actually doesn't
> work,
> > > presumably because of an interaction with breakpad, but a temporary
> > > directory is left on my filesystem, so that's nice.)
> > >
> > > -- Philip
> > >
> > > On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <ev...@isecc.com>
> > > wrote:
> > >
> > > > Is your cluster Kerberized at all, especially the Impala daemon - it
> > > > doesn’t seem to be enrolled in the KDC at all
> > > >
> > > > You / your personal account/principal is definitely enrolled though
> > > >
> > > > And there is definetly a KDC in your environment
> > > >
> > > > -----Original Message-----
> > > > From: Philip Zeyliger [mailto:philip@cloudera.com]
> > > > Sent: Tuesday, December 12, 2017 11:26 PM
> > > > To: dev@impala.apache.org
> > > > Subject: thrift-server-test
> > > >
> > > > Hi folks,
> > > >
> > > > I've been running into issues with thrift-server-test and Kerberos.
> > Below
> > > > is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-
> > > server-test";
> > > > both SslConnectivity/1 and
> > > > SslConnectivity/2 fail the same way.
> > > >
> > > > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> > > inside
> > > > of an Ubuntu 16.04 Docker container.
> > > >
> > > > Does this ring any bells?
> > > >
> > > > Thanks!
> > > >
> > > > -- Philip
> > > >
> > > >
> > > > [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > > SslConnectivity/2
> > > > Loading random data
> > > > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for
> > realm
> > > '
> > > > KRBTEST.COM',
> > > > master key name 'K/M@KRBTEST.COM'
> > > > [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > setting
> > > > up network...
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > > listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > > > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> > > > philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd
> 12:
> > > udp
> > > > ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> > > > krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> > > > philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > > commencing operation
> > > > krb5kdc: starting...
> > > > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > > > [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > WARNING: no policy specified for impala/localhost@KRBTEST.COM;
> > > defaulting
> > > > to no policy Principal "impala/localhost@KRBTEST.COM" created.
> > > > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > > > [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > Entry for principal impala/localhost with kvno 2, encryption type
> > > > aes256-cts-hmac-sha1-96 added to keytab
> > > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > > Entry for principal impala/localhost with kvno 2, encryption type
> > > > aes128-cts-hmac-sha1-96 added to keytab
> > > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > > Entry for principal impala/localhost with kvno 2, encryption type
> > > > des3-cbc-sha1 added to keytab
> > > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > > Entry for principal impala/localhost with kvno 2, encryption type
> > > > arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> > > > krb5kdc/impala_localhost.keytab.
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > AS_REQ
> > > > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime
> 1513120922,
> > > > etypes
> > > > {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> > > > KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't
> find
> > > > appropriate cache for server principal impala@localhost [31476]
> > > > 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
> > > > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > > > [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM
> ->
> > > > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > > > -1765328243/Matching credential not found [31476] 1513120922.532407:
> > > > Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/localhost@localhost
> > > > from FILE:/tmp/krb5cc_impala_internal with
> > > > result: -1765328243/Matching credential not found [31476]
> > > > 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/
> > > > KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> > > > result: 0/Success
> > > > [31476] 1513120922.532441: Starting with TGT for client realm:
> impala/
> > > > localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476]
> > > > 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> > > > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal
> with
> > > > result: -1765328243/Matching credential not found [31476]
> > > > 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using
> > TGT
> > > > krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491: Generated
> > > > subkey for TGS request: aes256-cts/005D [31476] 1513120922.532524:
> > etypes
> > > > requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1,
> > > rc4-hmac,
> > > > camellia128-cts, camellia256-cts [31476] 1513120922.532574: Encoding
> > > > request body and padata into FAST request [31476] 1513120922.532616:
> > > > Sending request (951 bytes) to KRBTEST.COM [31476]
> 1513120922.532630:
> > > > Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending
> initial
> > > UDP
> > > > request to dgram
> > > > 127.0.0.1:51781
> > > > [31586] 1513120922.532790: AP-REQ ticket: impala/
> localhost@KRBTEST.COM
> > > ->
> > > > krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586]
> > > > 1513120922.532814: Negotiated enctype based on authenticator:
> > > > aes256-cts
> > > > [31586] 1513120922.532820: Authenticator contains subkey:
> > aes256-cts/005D
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > TGS_REQ
> > > > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime
> 0,
> > > > impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM,
> Server
> > > not
> > > > found in Kerberos database [31476] 1513120922.533028: Received answer
> > > (491
> > > > bytes) from dgram
> > > > 127.0.0.1:51781
> > > > [31476] 1513120922.533044: Response was not from master KDC [31476]
> > > > 1513120922.533053: Decoding FAST response [31476] 1513120922.533081:
> > TGS
> > > > request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM not
> > > > found in Kerberos database
> > > > /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153:
> Failure
> > > > Value of: status_.ok()
> > > > Actual: false
> > > > Expected: true
> > > > Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> > > > failure: GSSAPI Error: Unspecified GSS failure. Minor code may
> provide
> > > > more information (Server krbtgt/localhost@KRBTEST.COM not found in
> > > > Kerberos
> > > > database))
> > > >
> > > > [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > > > SslConnectivity/2,
> > > > where GetParam() = 2 (100 ms)
> > > >
> > > >
> > >
> >
>
Re: thrift-server-test
Posted by Philip Zeyliger <ph...@cloudera.com>.
Hi Sailiesh,
Is this what you'd expect?
$klist /tmp/krb5cc_impala_internal
Ticket cache: FILE:/tmp/krb5cc_impala_internal
Default principal: impala/localhost@KRBTEST.COM
Valid starting Expires Service principal
01/08/2018 15:39:23 01/09/2018 15:39:23 krbtgt/KRBTEST.COM@KRBTEST.COM
renew until 01/15/2018 15:39:23
Thanks!
On Mon, Jan 8, 2018 at 12:20 PM, Sailesh Mukil <sa...@cloudera.com> wrote:
> Can you run the test again, and klist the contents of the credential cache
> and post the error logs again? Looks like "impala/localhost" might not be
> stored as expected in the cache on your machine.
>
> On Wed, Dec 13, 2017 at 2:47 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > The KDC in this case is the "minikdc" from
> > https://github.com/apache/impala/blob/master/be/src/
> > kudu/security/test/mini_kdc.cc.
> > I see evidence of it, and have been able to look at its configuration by,
> > um, adding --gtest_break_on_failure. (The feature actually doesn't work,
> > presumably because of an interaction with breakpad, but a temporary
> > directory is left on my filesystem, so that's nice.)
> >
> > -- Philip
> >
> > On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <ev...@isecc.com>
> > wrote:
> >
> > > Is your cluster Kerberized at all, especially the Impala daemon - it
> > > doesn’t seem to be enrolled in the KDC at all
> > >
> > > You / your personal account/principal is definitely enrolled though
> > >
> > > And there is definetly a KDC in your environment
> > >
> > > -----Original Message-----
> > > From: Philip Zeyliger [mailto:philip@cloudera.com]
> > > Sent: Tuesday, December 12, 2017 11:26 PM
> > > To: dev@impala.apache.org
> > > Subject: thrift-server-test
> > >
> > > Hi folks,
> > >
> > > I've been running into issues with thrift-server-test and Kerberos.
> Below
> > > is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-
> > server-test";
> > > both SslConnectivity/1 and
> > > SslConnectivity/2 fail the same way.
> > >
> > > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> > inside
> > > of an Ubuntu 16.04 Docker container.
> > >
> > > Does this ring any bells?
> > >
> > > Thanks!
> > >
> > > -- Philip
> > >
> > >
> > > [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > SslConnectivity/2
> > > Loading random data
> > > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for
> realm
> > '
> > > KRBTEST.COM',
> > > master key name 'K/M@KRBTEST.COM'
> > > [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> > > result: 0/Success
> > > [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> > > result: 0/Success
> > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > setting
> > > up network...
> > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> > > philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12:
> > udp
> > > ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> > > krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> > > philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > commencing operation
> > > krb5kdc: starting...
> > > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > > [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> > > result: 0/Success
> > > WARNING: no policy specified for impala/localhost@KRBTEST.COM;
> > defaulting
> > > to no policy Principal "impala/localhost@KRBTEST.COM" created.
> > > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > > [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> with
> > > result: 0/Success
> > > Entry for principal impala/localhost with kvno 2, encryption type
> > > aes256-cts-hmac-sha1-96 added to keytab
> > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > Entry for principal impala/localhost with kvno 2, encryption type
> > > aes128-cts-hmac-sha1-96 added to keytab
> > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > Entry for principal impala/localhost with kvno 2, encryption type
> > > des3-cbc-sha1 added to keytab
> > > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > > Entry for principal impala/localhost with kvno 2, encryption type
> > > arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> > > krb5kdc/impala_localhost.keytab.
> > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> AS_REQ
> > > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> > > etypes
> > > {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> > > KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't find
> > > appropriate cache for server principal impala@localhost [31476]
> > > 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
> > > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > > [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> > > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > > -1765328243/Matching credential not found [31476] 1513120922.532407:
> > > Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost
> > > from FILE:/tmp/krb5cc_impala_internal with
> > > result: -1765328243/Matching credential not found [31476]
> > > 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/
> > > KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> > > result: 0/Success
> > > [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> > > localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476]
> > > 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> > > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> > > result: -1765328243/Matching credential not found [31476]
> > > 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using
> TGT
> > > krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491: Generated
> > > subkey for TGS request: aes256-cts/005D [31476] 1513120922.532524:
> etypes
> > > requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1,
> > rc4-hmac,
> > > camellia128-cts, camellia256-cts [31476] 1513120922.532574: Encoding
> > > request body and padata into FAST request [31476] 1513120922.532616:
> > > Sending request (951 bytes) to KRBTEST.COM [31476] 1513120922.532630:
> > > Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending initial
> > UDP
> > > request to dgram
> > > 127.0.0.1:51781
> > > [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM
> > ->
> > > krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586]
> > > 1513120922.532814: Negotiated enctype based on authenticator:
> > > aes256-cts
> > > [31586] 1513120922.532820: Authenticator contains subkey:
> aes256-cts/005D
> > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > TGS_REQ
> > > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> > > impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server
> > not
> > > found in Kerberos database [31476] 1513120922.533028: Received answer
> > (491
> > > bytes) from dgram
> > > 127.0.0.1:51781
> > > [31476] 1513120922.533044: Response was not from master KDC [31476]
> > > 1513120922.533053: Decoding FAST response [31476] 1513120922.533081:
> TGS
> > > request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM not
> > > found in Kerberos database
> > > /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> > > Value of: status_.ok()
> > > Actual: false
> > > Expected: true
> > > Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> > > failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
> > > more information (Server krbtgt/localhost@KRBTEST.COM not found in
> > > Kerberos
> > > database))
> > >
> > > [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > > SslConnectivity/2,
> > > where GetParam() = 2 (100 ms)
> > >
> > >
> >
>
Re: thrift-server-test
Posted by Sailesh Mukil <sa...@cloudera.com>.
Can you run the test again, and klist the contents of the credential cache
and post the error logs again? Looks like "impala/localhost" might not be
stored as expected in the cache on your machine.
On Wed, Dec 13, 2017 at 2:47 PM, Philip Zeyliger <ph...@cloudera.com>
wrote:
> The KDC in this case is the "minikdc" from
> https://github.com/apache/impala/blob/master/be/src/
> kudu/security/test/mini_kdc.cc.
> I see evidence of it, and have been able to look at its configuration by,
> um, adding --gtest_break_on_failure. (The feature actually doesn't work,
> presumably because of an interaction with breakpad, but a temporary
> directory is left on my filesystem, so that's nice.)
>
> -- Philip
>
> On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <ev...@isecc.com>
> wrote:
>
> > Is your cluster Kerberized at all, especially the Impala daemon - it
> > doesn’t seem to be enrolled in the KDC at all
> >
> > You / your personal account/principal is definitely enrolled though
> >
> > And there is definetly a KDC in your environment
> >
> > -----Original Message-----
> > From: Philip Zeyliger [mailto:philip@cloudera.com]
> > Sent: Tuesday, December 12, 2017 11:26 PM
> > To: dev@impala.apache.org
> > Subject: thrift-server-test
> >
> > Hi folks,
> >
> > I've been running into issues with thrift-server-test and Kerberos. Below
> > is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-
> server-test";
> > both SslConnectivity/1 and
> > SslConnectivity/2 fail the same way.
> >
> > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> inside
> > of an Ubuntu 16.04 Docker container.
> >
> > Does this ring any bells?
> >
> > Thanks!
> >
> > -- Philip
> >
> >
> > [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2
> > Loading random data
> > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm
> '
> > KRBTEST.COM',
> > master key name 'K/M@KRBTEST.COM'
> > [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> setting
> > up network...
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> > philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12:
> udp
> > ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> > krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> > philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > commencing operation
> > krb5kdc: starting...
> > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > WARNING: no policy specified for impala/localhost@KRBTEST.COM;
> defaulting
> > to no policy Principal "impala/localhost@KRBTEST.COM" created.
> > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes256-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes128-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > des3-cbc-sha1 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> > krb5kdc/impala_localhost.keytab.
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> > etypes
> > {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> > KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't find
> > appropriate cache for server principal impala@localhost [31476]
> > 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
> > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > -1765328243/Matching credential not found [31476] 1513120922.532407:
> > Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost
> > from FILE:/tmp/krb5cc_impala_internal with
> > result: -1765328243/Matching credential not found [31476]
> > 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/
> > KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> > result: 0/Success
> > [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> > localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476]
> > 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> > result: -1765328243/Matching credential not found [31476]
> > 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using TGT
> > krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491: Generated
> > subkey for TGS request: aes256-cts/005D [31476] 1513120922.532524: etypes
> > requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1,
> rc4-hmac,
> > camellia128-cts, camellia256-cts [31476] 1513120922.532574: Encoding
> > request body and padata into FAST request [31476] 1513120922.532616:
> > Sending request (951 bytes) to KRBTEST.COM [31476] 1513120922.532630:
> > Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending initial
> UDP
> > request to dgram
> > 127.0.0.1:51781
> > [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM
> ->
> > krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586]
> > 1513120922.532814: Negotiated enctype based on authenticator:
> > aes256-cts
> > [31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> TGS_REQ
> > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> > impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server
> not
> > found in Kerberos database [31476] 1513120922.533028: Received answer
> (491
> > bytes) from dgram
> > 127.0.0.1:51781
> > [31476] 1513120922.533044: Response was not from master KDC [31476]
> > 1513120922.533053: Decoding FAST response [31476] 1513120922.533081: TGS
> > request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM not
> > found in Kerberos database
> > /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> > Value of: status_.ok()
> > Actual: false
> > Expected: true
> > Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> > failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
> > more information (Server krbtgt/localhost@KRBTEST.COM not found in
> > Kerberos
> > database))
> >
> > [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > SslConnectivity/2,
> > where GetParam() = 2 (100 ms)
> >
> >
>
Re: thrift-server-test
Posted by Philip Zeyliger <ph...@cloudera.com>.
The KDC in this case is the "minikdc" from
https://github.com/apache/impala/blob/master/be/src/kudu/security/test/mini_kdc.cc.
I see evidence of it, and have been able to look at its configuration by,
um, adding --gtest_break_on_failure. (The feature actually doesn't work,
presumably because of an interaction with breakpad, but a temporary
directory is left on my filesystem, so that's nice.)
-- Philip
On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <ev...@isecc.com> wrote:
> Is your cluster Kerberized at all, especially the Impala daemon - it
> doesn’t seem to be enrolled in the KDC at all
>
> You / your personal account/principal is definitely enrolled though
>
> And there is definetly a KDC in your environment
>
> -----Original Message-----
> From: Philip Zeyliger [mailto:philip@cloudera.com]
> Sent: Tuesday, December 12, 2017 11:26 PM
> To: dev@impala.apache.org
> Subject: thrift-server-test
>
> Hi folks,
>
> I've been running into issues with thrift-server-test and Kerberos. Below
> is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-server-test";
> both SslConnectivity/1 and
> SslConnectivity/2 fail the same way.
>
> I'm running Ubuntu16.04. I've seen this both on my host, as well as inside
> of an Ubuntu 16.04 Docker container.
>
> Does this ring any bells?
>
> Thanks!
>
> -- Philip
>
>
> [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
> Loading random data
> Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
> KRBTEST.COM',
> master key name 'K/M@KRBTEST.COM'
> [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): setting
> up network...
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12: udp
> ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> philip-dev.gce.cloudera.com krb5kdc[31586](info):
> commencing operation
> krb5kdc: starting...
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> WARNING: no policy specified for impala/localhost@KRBTEST.COM; defaulting
> to no policy Principal "impala/localhost@KRBTEST.COM" created.
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> Entry for principal impala/localhost with kvno 2, encryption type
> aes256-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> aes128-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> des3-cbc-sha1 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> krb5kdc/impala_localhost.keytab.
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> etypes
> {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't find
> appropriate cache for server principal impala@localhost [31476]
> 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
> -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> -1765328243/Matching credential not found [31476] 1513120922.532407:
> Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost
> from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found [31476]
> 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/
> KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> result: 0/Success
> [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476]
> 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found [31476]
> 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using TGT
> krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491: Generated
> subkey for TGS request: aes256-cts/005D [31476] 1513120922.532524: etypes
> requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac,
> camellia128-cts, camellia256-cts [31476] 1513120922.532574: Encoding
> request body and padata into FAST request [31476] 1513120922.532616:
> Sending request (951 bytes) to KRBTEST.COM [31476] 1513120922.532630:
> Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending initial UDP
> request to dgram
> 127.0.0.1:51781
> [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM ->
> krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586]
> 1513120922.532814: Negotiated enctype based on authenticator:
> aes256-cts
> [31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): TGS_REQ
> (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server not
> found in Kerberos database [31476] 1513120922.533028: Received answer (491
> bytes) from dgram
> 127.0.0.1:51781
> [31476] 1513120922.533044: Response was not from master KDC [31476]
> 1513120922.533053: Decoding FAST response [31476] 1513120922.533081: TGS
> request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM not
> found in Kerberos database
> /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
> more information (Server krbtgt/localhost@KRBTEST.COM not found in
> Kerberos
> database))
>
> [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2,
> where GetParam() = 2 (100 ms)
>
>
RE: thrift-server-test
Posted by Evo Eftimov <ev...@isecc.com>.
Is your cluster Kerberized at all, especially the Impala daemon - it doesn’t seem to be enrolled in the KDC at all
You / your personal account/principal is definitely enrolled though
And there is definetly a KDC in your environment
-----Original Message-----
From: Philip Zeyliger [mailto:philip@cloudera.com]
Sent: Tuesday, December 12, 2017 11:26 PM
To: dev@impala.apache.org
Subject: thrift-server-test
Hi folks,
I've been running into issues with thrift-server-test and Kerberos. Below is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
SslConnectivity/2 fail the same way.
I'm running Ubuntu16.04. I've seen this both on my host, as well as inside of an Ubuntu 16.04 Docker container.
Does this ring any bells?
Thanks!
-- Philip
[ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
Loading random data
Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
[31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
[31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): setting up network...
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12: udp ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
commencing operation
krb5kdc: starting...
Authenticating as principal philip/admin@KRBTEST.COM with password.
[31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
WARNING: no policy specified for impala/localhost@KRBTEST.COM; defaulting to no policy Principal "impala/localhost@KRBTEST.COM" created.
Authenticating as principal philip/admin@KRBTEST.COM with password.
[31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Entry for principal impala/localhost with kvno 2, encryption type
aes256-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
aes128-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
des3-cbc-sha1 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922, etypes
{rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/ KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532304: ccselect can't find appropriate cache for server principal impala@localhost [31476] 1513120922.532347: Getting credentials impala/localhost@KRBTEST.COM
-> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
[31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM -> impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
-1765328243/Matching credential not found [31476] 1513120922.532407: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found [31476] 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
result: 0/Success
[31476] 1513120922.532441: Starting with TGT for client realm: impala/ localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM -> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found [31476] 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM using TGT krbtgt/KRBTEST.COM@KRBTEST.COM [31476] 1513120922.532491: Generated subkey for TGS request: aes256-cts/005D [31476] 1513120922.532524: etypes requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts [31476] 1513120922.532574: Encoding request body and padata into FAST request [31476] 1513120922.532616: Sending request (951 bytes) to KRBTEST.COM [31476] 1513120922.532630: Resolving hostname 127.0.0.1 [31476] 1513120922.532648: Sending initial UDP request to dgram
127.0.0.1:51781
[31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F [31586] 1513120922.532814: Negotiated enctype based on authenticator:
aes256-cts
[31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): TGS_REQ
(6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0, impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server not found in Kerberos database [31476] 1513120922.533028: Received answer (491 bytes) from dgram
127.0.0.1:51781
[31476] 1513120922.533044: Response was not from master KDC [31476] 1513120922.533053: Decoding FAST response [31476] 1513120922.533081: TGS request result: -1765328377/Server krbtgt/ localhost@KRBTEST.COM not found in Kerberos database
/home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure Value of: status_.ok()
Actual: false
Expected: true
Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/localhost@KRBTEST.COM not found in Kerberos
database))
[ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2,
where GetParam() = 2 (100 ms)
Re: thrift-server-test
Posted by Philip Zeyliger <ph...@cloudera.com>.
$krb5-config --version
Kerberos 5 release 1.13.2
On Tue, Dec 12, 2017 at 3:39 PM, Michael Ho <kw...@cloudera.com> wrote:
> Not that I know the answer to the problem you are hitting. Just wondering
> what version of Kerberos library (krb5-config --version) are you running ?
>
> On Tue, Dec 12, 2017 at 3:25 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > Hi folks,
> >
> > I've been running into issues with thrift-server-test and Kerberos. Below
> > is an excerpt of "KRB5_TRACE=/dev/stderr
> > be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
> > SslConnectivity/2 fail the same way.
> >
> > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> inside
> > of an Ubuntu 16.04 Docker container.
> >
> > Does this ring any bells?
> >
> > Thanks!
> >
> > -- Philip
> >
> >
> > [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2
> > Loading random data
> > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm
> '
> > KRBTEST.COM',
> > master key name 'K/M@KRBTEST.COM'
> > [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> setting
> > up network...
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening
> > on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening
> > on fd 12: udp ::.51781 (pktinfo)
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set
> up 2
> > sockets
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > commencing operation
> > krb5kdc: starting...
> > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > WARNING: no policy specified for impala/localhost@KRBTEST.COM;
> defaulting
> > to no policy
> > Principal "impala/localhost@KRBTEST.COM" created.
> > Authenticating as principal philip/admin@KRBTEST.COM with password.
> > [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes256-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes128-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > des3-cbc-sha1 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > arcfour-hmac added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> > (6
> > etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> etypes
> > {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> > KRBTEST.COM@KRBTEST.COM
> > [31476] 1513120922.532304: ccselect can't find appropriate cache for
> server
> > principal impala@localhost
> > [31476] 1513120922.532347: Getting credentials impala/
> > localhost@KRBTEST.COM
> > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > -1765328243/Matching credential not found
> > [31476] 1513120922.532407: Retrieving impala/localhost@KRBTEST.COM ->
> > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> > result: -1765328243/Matching credential not found
> > [31476] 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM ->
> > krbtgt/KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal
> with
> > result: 0/Success
> > [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> > localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM
> > [31476] 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> > result: -1765328243/Matching credential not found
> > [31476] 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM
> > using TGT krbtgt/KRBTEST.COM@KRBTEST.COM
> > [31476] 1513120922.532491: Generated subkey for TGS request:
> > aes256-cts/005D
> > [31476] 1513120922.532524: etypes requested in TGS request: aes256-cts,
> > aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts
> > [31476] 1513120922.532574: Encoding request body and padata into FAST
> > request
> > [31476] 1513120922.532616: Sending request (951 bytes) to KRBTEST.COM
> > [31476] 1513120922.532630: Resolving hostname 127.0.0.1
> > [31476] 1513120922.532648: Sending initial UDP request to dgram
> > 127.0.0.1:51781
> > [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM
> ->
> > krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F
> > [31586] 1513120922.532814: Negotiated enctype based on authenticator:
> > aes256-cts
> > [31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> TGS_REQ
> > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> > impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server
> not
> > found in Kerberos database
> > [31476] 1513120922.533028: Received answer (491 bytes) from dgram
> > 127.0.0.1:51781
> > [31476] 1513120922.533044: Response was not from master KDC
> > [31476] 1513120922.533053: Decoding FAST response
> > [31476] 1513120922.533081: TGS request result: -1765328377/Server krbtgt/
> > localhost@KRBTEST.COM not found in Kerberos database
> > /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> > Value of: status_.ok()
> > Actual: false
> > Expected: true
> > Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> > failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
> > more information (Server krbtgt/localhost@KRBTEST.COM not found in
> > Kerberos
> > database))
> >
> > [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > SslConnectivity/2,
> > where GetParam() = 2 (100 ms)
> >
>
>
>
> --
> Thanks,
> Michael
>
Re: thrift-server-test
Posted by Michael Ho <kw...@cloudera.com>.
Not that I know the answer to the problem you are hitting. Just wondering
what version of Kerberos library (krb5-config --version) are you running ?
On Tue, Dec 12, 2017 at 3:25 PM, Philip Zeyliger <ph...@cloudera.com>
wrote:
> Hi folks,
>
> I've been running into issues with thrift-server-test and Kerberos. Below
> is an excerpt of "KRB5_TRACE=/dev/stderr
> be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
> SslConnectivity/2 fail the same way.
>
> I'm running Ubuntu16.04. I've seen this both on my host, as well as inside
> of an Ubuntu 16.04 Docker container.
>
> Does this ring any bells?
>
> Thanks!
>
> -- Philip
>
>
> [ RUN ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
> Loading random data
> Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
> KRBTEST.COM',
> master key name 'K/M@KRBTEST.COM'
> [31585] 1513120922.459517: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> [31586] 1513120922.472314: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): setting
> up network...
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> listening
> on fd 11: udp 0.0.0.0.51781 (pktinfo)
> krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> listening
> on fd 12: udp ::.51781 (pktinfo)
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set up 2
> sockets
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> commencing operation
> krb5kdc: starting...
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31589] 1513120922.498913: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> WARNING: no policy specified for impala/localhost@KRBTEST.COM; defaulting
> to no policy
> Principal "impala/localhost@KRBTEST.COM" created.
> Authenticating as principal philip/admin@KRBTEST.COM with password.
> [31590] 1513120922.508777: Retrieving K/M@KRBTEST.COM from
> FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> result: 0/Success
> Entry for principal impala/localhost with kvno 2, encryption type
> aes256-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> aes128-cts-hmac-sha1-96 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> des3-cbc-sha1 added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Entry for principal impala/localhost with kvno 2, encryption type
> arcfour-hmac added to keytab
> WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> (6
> etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922, etypes
> {rep=18 tkt=18 ses=18}, impala/localhost@KRBTEST.COM for krbtgt/
> KRBTEST.COM@KRBTEST.COM
> [31476] 1513120922.532304: ccselect can't find appropriate cache for server
> principal impala@localhost
> [31476] 1513120922.532347: Getting credentials impala/
> localhost@KRBTEST.COM
> -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> [31476] 1513120922.532382: Retrieving impala/localhost@KRBTEST.COM ->
> impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> -1765328243/Matching credential not found
> [31476] 1513120922.532407: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found
> [31476] 1513120922.532433: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/KRBTEST.COM@KRBTEST.COM from FILE:/tmp/krb5cc_impala_internal with
> result: 0/Success
> [31476] 1513120922.532441: Starting with TGT for client realm: impala/
> localhost@KRBTEST.COM -> krbtgt/KRBTEST.COM@KRBTEST.COM
> [31476] 1513120922.532467: Retrieving impala/localhost@KRBTEST.COM ->
> krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> result: -1765328243/Matching credential not found
> [31476] 1513120922.532475: Requesting TGT krbtgt/localhost@KRBTEST.COM
> using TGT krbtgt/KRBTEST.COM@KRBTEST.COM
> [31476] 1513120922.532491: Generated subkey for TGS request:
> aes256-cts/005D
> [31476] 1513120922.532524: etypes requested in TGS request: aes256-cts,
> aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts
> [31476] 1513120922.532574: Encoding request body and padata into FAST
> request
> [31476] 1513120922.532616: Sending request (951 bytes) to KRBTEST.COM
> [31476] 1513120922.532630: Resolving hostname 127.0.0.1
> [31476] 1513120922.532648: Sending initial UDP request to dgram
> 127.0.0.1:51781
> [31586] 1513120922.532790: AP-REQ ticket: impala/localhost@KRBTEST.COM ->
> krbtgt/KRBTEST.COM@KRBTEST.COM, session key aes256-cts/580F
> [31586] 1513120922.532814: Negotiated enctype based on authenticator:
> aes256-cts
> [31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
> Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): TGS_REQ
> (6 etypes {18 17 16 23 25 26}) 127.0.0.1: UNKNOWN_SERVER: authtime 0,
> impala/localhost@KRBTEST.COM for krbtgt/localhost@KRBTEST.COM, Server not
> found in Kerberos database
> [31476] 1513120922.533028: Received answer (491 bytes) from dgram
> 127.0.0.1:51781
> [31476] 1513120922.533044: Response was not from master KDC
> [31476] 1513120922.533053: Decoding FAST response
> [31476] 1513120922.533081: TGS request result: -1765328377/Server krbtgt/
> localhost@KRBTEST.COM not found in Kerberos database
> /home/philip/src/impala/be/src/rpc/thrift-server-test.cc:153: Failure
> Value of: status_.ok()
> Actual: false
> Expected: true
> Error: Couldn't open transport for localhost:62119 (SASL(-1): generic
> failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide
> more information (Server krbtgt/localhost@KRBTEST.COM not found in
> Kerberos
> database))
>
> [ FAILED ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2,
> where GetParam() = 2 (100 ms)
>
--
Thanks,
Michael