[ozone][security] issue when trying ozone scm with kerberos

[Diego Jaramillo <dj...@gmail.com>]

Hello ozone community

I'm getting the following error when trying to run any action in SCM with
kerberos enabled:

Caused by: java.io.IOException: Couldn't setup connection for
myuser@EXAMPLE.COM to scm-server.example.com/10.XXX.YYY.ZZZ:9961
        ...
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed

when I check the SCM logs and the service starts properly
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/
scm-server.example.com@EXAMPLE.COM
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@24a1c17f
{logs,/logs,file:///var/log/ozone/,AVAILABLE}
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.s.ServletContextHandler@532721fd
{static,/static,jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/static,AVAILABLE}
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Using keytab /opt/ozone/keytabs/spnego.keytab, for principal HTTP/
scm-server.example.com@EXAMPLE.COM
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.server.handler.ContextHandler: Started
o.e.j.w.WebAppContext@62ddd21b
{scm,/,file:///tmp/jetty-0_0_0_0-9877-hadoop-hdds-server-scm-1_0_0_jar-_-any-4975172152496600541.dir/webapp/,AVAILABLE}{jar:file:/opt/ozone/share/ozone/lib/hadoop-hdds-server-scm-1.0.0.jar!/webapps/scm}
[Listener at 0.0.0.0/9860] INFO
org.eclipse.jetty.util.ssl.SslContextFactory: x509=X509@783efb48(
scm-server.example.com,h=[scm-server.example.com, scm-server],w=[]) for
Server@c446b14
[provider=null,keyStore=file:///opt/ozone/jks/scm-server.jks,trustStore=file:///opt/ozone/jks/cacerts]
[Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.AbstractConnector:
Started ServerConnector@13006998{SSL,[ssl, http/1.1]}{0.0.0.0:9877}
[Listener at 0.0.0.0/9860] INFO org.eclipse.jetty.server.Server: Started
@2301ms
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink prometheus started
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Registered sink
prometheus
[Listener at 0.0.0.0/9860] INFO
org.apache.hadoop.hdds.server.http.BaseHttpServer: HTTPS server of scm
listening at https://0.0.0.0:9877

and I only see this message:
[Socket Reader #1 for port 9961] WARN
SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for
10.XXX.YYY.ZZZ:52306:null
(GSS initiate failed) with true cause: (GSS initiate failed)

I used this template to build my configs:
https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config

my auth_to_local currently is like this:
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](.*)s/.*/scm/
DEFAULT</value>
  </property>

I already tested the rules and they seems to be OK:
$ hadoop kerbname myuser@EXAMPLE.COM
Name: myuser@EXAMPLE.COM to myuser
$ hadoop kerbname scm/scm-server.example.com@EXAMPLE.COM
Name: scm/scm-server.example.com@EXAMPLE.COM to scm
Also tried the following auth_to_local, but the result is the same:
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE:[1:$1@$0](scm@EXAMPLE.COM)s/.*/scm/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0](.*)s/.*/root/
DEFAULT</value>
  </property>

Have you had this issue before? Could you provide any suggestion for this?
I'm also attaching as reference my templates for the core-site, ozone-site
and hdfs-site.

I'm using ozone 1.0 downloaded from the site
build =
https://github.com/apache/hadoop-ozone.git/28d372ca903b4741131bace09e0339e9161257bb
; compiled by 'sammi' on 2020-08-25T13:04Z
java = 1.8.0_201

But it is the same issue when I build it from source.

Thank you in advance.

Diego