You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/04/01 08:28:05 UTC
[GitHub] [spark] gaborgsomogyi commented on issue #24204: [SPARK-27270][SS]
Add Kafka dynamic JAAS authentication debug possibility
gaborgsomogyi commented on issue #24204: [SPARK-27270][SS] Add Kafka dynamic JAAS authentication debug possibility
URL: https://github.com/apache/spark/pull/24204#issuecomment-478485095
Maybe my explanation was not enough/clean. Let me give a little bit more details.
I've created a small standalone application where these things can be tested easily.
The application connect to a secure Kafka cluster and tries to do authentication with dynamic JAAS configuration (where `debug=true` can be set). I've re-tested everything to give exact logs.
Test with `debug=false`:
Command: `java -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true ...`
Output:
```
Java config name: null
Native config name: /etc/krb5.conf
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 70; type: 16
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 62; type: 23
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 8
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 3
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 1
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 1
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
>>> KdcAccessibility: reset
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23.
>>> KrbAsReq creating message
getKDCFromDNS using UDP
getKDCFromDNS using TCP
>>> KrbKdcReq send: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000, number of retries =3, #bytes=148
>>> KDCCommunication: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000,Attempt =1, #bytes=148
>>> KrbKdcReq send: #bytes read=775
>>> KdcAccessibility: remove gsomogyi-cdh6x-secure-1.gce.cloudera.com
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbAsRep cons in KrbAsReq.getReply systest
>>> 19/04/01 10:08:56 INFO authenticator.AbstractLogin: Successfully logged in.
>>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT refresh thread started.
>>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT valid starting at: Mon Apr 01 10:08:55 CEST 2019
>>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT expires: Wed May 01 10:08:55 CEST 2019
>>> 19/04/01 10:08:56 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT refresh sleeping until: Fri Apr 26 05:03:45 CEST 2019
>>> 19/04/01 10:08:56 INFO utils.AppInfoParser: Kafka version : 2.0.0
>>> 19/04/01 10:08:56 INFO utils.AppInfoParser: Kafka commitId : 3402a8361b734732
```
Test with `debug=true`:
Command: `java ...`
Output:
```
Java config name: null
Native config name: /etc/krb5.conf
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /Users/gaborsomogyi/kafka-delegation-token/kafka-consumer/systest.keytab refreshKrb5Config is false principal is systest@GCE.CLOUDERA.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 70; type: 16
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 62; type: 23
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 8
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 3
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 1
>>> KeyTabInputStream, readName(): GCE.CLOUDERA.COM
>>> KeyTabInputStream, readName(): systest
>>> KeyTab: load() entry length: 54; type: 1
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
>>> KdcAccessibility: reset
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 17 16 23.
>>> KrbAsReq creating message
getKDCFromDNS using UDP
getKDCFromDNS using TCP
>>> KrbKdcReq send: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000, number of retries =3, #bytes=148
>>> KDCCommunication: kdc=gsomogyi-cdh6x-secure-1.gce.cloudera.com UDP:88, timeout=30000,Attempt =1, #bytes=148
>>> KrbKdcReq send: #bytes read=775
>>> KdcAccessibility: remove gsomogyi-cdh6x-secure-1.gce.cloudera.com
Looking for keys for: systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (1) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (3) for systest@GCE.CLOUDERA.COM
Found unsupported keytype (8) for systest@GCE.CLOUDERA.COM
Added key: 23version: 1
Added key: 16version: 1
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbAsRep cons in KrbAsReq.getReply systest
principal is systest@GCE.CLOUDERA.COM
Will use keytab
Commit Succeeded
>>> 19/04/01 10:11:25 INFO authenticator.AbstractLogin: Successfully logged in.
>>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT refresh thread started.
>>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT valid starting at: Mon Apr 01 10:11:24 CEST 2019
>>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT expires: Wed May 01 10:11:24 CEST 2019
>>> 19/04/01 10:11:25 INFO kerberos.KerberosLogin: [Principal=systest@GCE.CLOUDERA.COM]: TGT refresh sleeping until: Fri Apr 26 02:37:27 CEST 2019
>>> 19/04/01 10:11:25 INFO utils.AppInfoParser: Kafka version : 2.0.0
>>> 19/04/01 10:11:25 INFO utils.AppInfoParser: Kafka commitId : 3402a8361b734732
```
As a final conclusion even if global krb debug flag is enabled `Krb5LoginModule` debug messages not shown. Please see from the first execution the following message is missing:
```
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /Users/gaborsomogyi/kafka-delegation-token/kafka-consumer/systest.keytab refreshKrb5Config is false principal is systest@GCE.CLOUDERA.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false
...
principal is systest@GCE.CLOUDERA.COM
Will use keytab
Commit Succeeded
...
```
These messages is just a sample and `Krb5LoginModule` provides much more debug information which may be helpful for debugging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org