Accumulo and Hadoop upgrades failing Kerberos authentication after 24 hours

["Sparks, Alex" <al...@cgi.com>]

Hi All,

We are trying to make major version upgrades of Accumulo and Hadoop:
Accumulo: 1.8.1 --> 2.0.1
Hadoop: 2.8.2 --> 3.0.3

They run as single non-clustered containers in a Docker environment. We have a separate Zookeeper container running which they talk to at version 3.4.10. Other client containers authenticate with Kerberos in order to retrieve information from Accumulo. We have a KDC running on a separate authentication VM which is reachable by both containers.

Once upgrading their major versions we get a problem with their Kerberos authentication. On initialisation, Accumulo and Hadoop run kinit commands to generate themselves ticket-granting-tickets (TGTs) which are valid for 24h, and the full application works as expected. After 24h, however, our client containers can no longer authenticate and access information from Accumulo, despite the clients having valid service tickets for Accumulo.
If we manually regenerate the TGT within Accumulo with another kinit command the problem still persists.
If we manually change the KDC's configuration to issue 10 minute tickets rather than 24h, then authentication breaks after 10 minutes regardless of each component's krb5.conf file.
Pre-upgrade all our containers must have been able to retrieve new tickets once their old ones had expired, but this no longer seems to be the case. The only way to fix the problem is by restarting the containers. Below are the configuration files for the various containers - any variables surrounded by "@" or "<>" are substituted in at runtime and point to valid paths / files / values.

Accumulo accumulo.properties:
general.kerberos.keytab=@KRB_KEYTAB@<mailto:general.kerberos.keytab=@KRB_KEYTAB@>
general.kerberos.principal=@KRB_PRINCIPAL@<mailto:general.kerberos.principal=@KRB_PRINCIPAL@>
instance.rpc.sasl.enabled=true
instance.secret=@SECRET@<mailto:instance.secret=@SECRET@>
instance.security.authenticator=org.apache.accumulo.server.security.handler.KerberosAuthenticator
instance.security.authorizor=org.apache.accumulo.server.security.handler.KerberosAuthorizor
instance.security.permissionHandler=org.apache.accumulo.server.security.handler.KerberosPermissionHandler
instance.volumes=@HDFS_VOLUMES@<mailto:instance.volumes=@HDFS_VOLUMES@>
instance.zookeeper.host=@ZOOKEEPERS@<mailto:instance.zookeeper.host=@ZOOKEEPERS@>
rpc.sasl.qop=auth
trace.token.property.keytab=@KRB_KEYTAB@<mailto:trace.token.property.keytab=@KRB_KEYTAB@>
trace.token.type=org.apache.accumulo.core.client.security.tokens.KerberosToken
trace.user=@KRB_PRINCIPAL@<mailto:trace.user=@KRB_PRINCIPAL@>
tserver.cache.data.size=@CACHE_DATA_SIZE@<mailto:tserver.cache.data.size=@CACHE_DATA_SIZE@>
tserver.cache.index.size=@CACHE_INDEX_SIZE@<mailto:tserver.cache.index.size=@CACHE_INDEX_SIZE@>
tserver.memory.maps.max=@MEMORY_MAPS_MAX@<mailto:tserver.memory.maps.max=@MEMORY_MAPS_MAX@>
tserver.memory.maps.native.enabled=false
tserver.sort.buffer.size=@SORT_BUFFER_SIZE@<mailto:tserver.sort.buffer.size=@SORT_BUFFER_SIZE@>
tserver.walog.max.size=@WALOG_MAX_SIZE@<mailto:tserver.walog.max.size=@WALOG_MAX_SIZE@>


Accumulo accumulo-client.properties:

instance.name=accumulo

instance.zookeepers=@ZOOKEEPERS@<mailto:instance.zookeepers=@ZOOKEEPERS@>

instance.zookeepers.timeout=30s



auth.type=kerberos

auth.principal=@KRB_PRINCIPAL@<mailto:auth.principal=@KRB_PRINCIPAL@>

auth.token=@KRB_KEYTAB@<mailto:auth.token=@KRB_KEYTAB@>



sasl.enabled=true

sasl.qop=auth

sasl.kerberos.server.primary=accumulo





Accumulo / Hadoop / Zookeeper krb5.conf (all identical):

includedir /etc/krb5.conf.d/



[logging]

 default = FILE:/var/log/krb5libs.log

 kdc = FILE:/var/log/krb5kdc.log

 admin_server = FILE:/var/log/kadmind.log



[libdefaults]

 dns_lookup_realm = false

 ticket_lifetime = 24h

 renew_lifetime = 7d

 forwardable = true

 rdns = false

 pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt

 default_realm = NRAC.UK

#default_ccache_name = KEYRING:persistent:%{uid}

 udp_preference_limit = 1



[realms]

 NRAC.UK = {

  kdc = <ldap-server>

  admin_server = <ldap-server>

  default_domain = <our_domain>

  database_module = openldap_ldapconf

 }



[domain_realm]

 .<our_domain> = <OUR_DOMAIN>

 <our_domain> = <OUR_DOMAIN>



[dbdefaults]

 ldap_kerberos_container_dn = cn=krbContainer,dc=nrac,dc=uk



[dbmodules]

 openldap_ldapconf = {

  db_library = kldap

  ldap_kdc_dn = "cn=nrac-ldapadm,dc=nrac,dc=uk"

  ldap_kadmind_dn = "cn=nrac-ldapadm,dc=nrac,dc=uk"

  ldap_service_password_file = /etc/krb5kdc/service.keyfile

  ldap_servers = ldaps://<ldap-server>

  ldap_conns_per_server = 5

 }



Any help would be much appreciated, many thanks.

Alex Sparks





Public