You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Jones, Oliver" <ol...@cgi.com> on 2019/03/26 20:14:16 UTC

Accumulo/Zookeeper Kerberos Integration

Hello all,

I sent this in a couple of months ago, but I'm not sure if I was subscribed to the mailing list correctly and didn't see any replies - so I thought I'd try again.

I'm having trouble getting Kerberos authentication to work between Zookeeper and Accumulo. I am not using any supporting platforms (e.g. Cloudera, Hortonworks) - this is all being done using Docker with single Accumulo (1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their own Docker network. My KDC is running on a separate CentOS machine, but can be reached by all of them. I have already managed to integrate Kerberos authentication with Hadoop and Accumulo, but cannot add Zookeeper client into the mix.


*        The Zookeeper container has this configuration:

/conf/zoo.cfg

# Kerberos Configuration
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

/conf/zookeeper_jaas_server.conf

Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/zookeeper-server.keytab"
principal="zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM<ma...@EXAMPLE.COM>";
};

export SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_server.conf

/conf/zookeeper_jaas_client.conf

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/zookeeper-client.keytab"
principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM<ma...@EXAMPLE.COM>";
};

export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_client.conf


*        And I've added this configuration to Accumulo:

/usr/local/zookeeper/conf/zookeeper_jaas_client.conf

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/accumulo.keytab"
principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM<ma...@EXAMPLE.COM>";
};

export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf"

This means that Zookeeper server starts by authenticating as principle zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM<ma...@EXAMPLE.COM> and Accumulo (a Zookeeper client) authenticates with Zookeeper using the zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM<ma...@EXAMPLE.COM> principle.

When starting Accumulo for the first time and running through an initialisation, this authentication works fine and Accumulo starts as expected. But when you restart Accumulo, the master fails to start with this error:

2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor] ERROR: Saw more than one ACL on the node
2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor] ERROR: Expected /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} , 31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=}
]
2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception, exiting
java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper is not protected.
        at org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDistributor.initialize(ZooAuthenticationKeyDistributor.java:86)
        at org.apache.accumulo.master.Master.run(Master.java:1223)
        at org.apache.accumulo.master.Master.main(Master.java:1434)
        at org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.java:33)
        at org.apache.accumulo.start.Main$1.run(Main.java:120)
        at java.lang.Thread.run(Thread.java:748)

It looks like the information that Accumulo writes to Zookeeper during the initialisation has two ACLs associated to it - one for the zookeeper-client Kerberos principle (sasl) and one for the Accumulo secret (digest). These two ACLs seem to fail one of the master's start-up checks and causes it to exit.

Is there anyway of disabling the Accumulo secret so that I only have the Kerberos ACL? Or is there something wrong with the way I've tried to implement this that would cause this problem?

I'd be very appreciative of any assistance.

Many thanks,

Oliver Jones

Re: Accumulo/Zookeeper Kerberos Integration

Posted by Josh Elser <el...@apache.org>.
By default, ZooKeeper allows fallback to read-only operations for 
clients, even when you have the SASL authentication provider set up. 
This is ultimately what lets Accumulo work.

Like Christopher says, the work to use SASL ACLs instead of DIGEST ACLs 
for ZK is waiting for someone to pick it up. You're not missing out on 
anything big in terms of the total security package (the model that 
Accumulo uses to prevent "rogue" TServers from joining the quorum 
without setting up Kerberos auth is sufficient), but it is a gap if you 
want everything using Kerberos.

This is all to clarify that Accumulo _can_ work when ZK is configured 
with SASL/Kerberos, to avoid people misconstruing things later :)

On 3/27/19 11:34 AM, Jones, Oliver wrote:
> Thank you for your response Christopher. I was unaware of that - I thought Accumulo could interacted with a kerberised Zookeeper. I will keep checking new releases to see if this changes.
> 
> Thanks,
> 
> Ollie
> 
> -----Original Message-----
> From: Christopher [mailto:ctubbsii@apache.org]
> Sent: 27 March 2019 02:12
> To: accumulo-user <us...@accumulo.apache.org>
> Subject: Re: Accumulo/Zookeeper Kerberos Integration
> 
> Accumulo does not currently support connecting to ZooKeeper using Kerberos as an alternative to the digest secret. Such a feature would need to be implemented.
> 
> On Tue, Mar 26, 2019 at 4:15 PM Jones, Oliver <ol...@cgi.com> wrote:
>>
>> Hello all,
>>
>>
>>
>> I sent this in a couple of months ago, but I’m not sure if I was subscribed to the mailing list correctly and didn’t see any replies – so I thought I’d try again.
>>
>>
>>
>> I’m having trouble getting Kerberos authentication to work between Zookeeper and Accumulo. I am not using any supporting platforms (e.g. Cloudera, Hortonworks) - this is all being done using Docker with single Accumulo (1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their own Docker network. My KDC is running on a separate CentOS machine, but can be reached by all of them. I have already managed to integrate Kerberos authentication with Hadoop and Accumulo, but cannot add Zookeeper client into the mix.
>>
>>
>>
>> ·        The Zookeeper container has this configuration:
>>
>>
>>
>> /conf/zoo.cfg
>>
>>
>>
>> # Kerberos Configuration
>>
>> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProv
>> ider
>>
>> kerberos.removeHostFromPrincipal=true
>>
>> kerberos.removeRealmFromPrincipal=true
>>
>> requireClientAuthScheme=sasl
>>
>> jaasLoginRenew=3600000
>>
>>
>>
>> /conf/zookeeper_jaas_server.conf
>>
>>
>>
>> Server {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> storeKey=true
>>
>> useTicketCache=false
>>
>> keyTab="/etc/security/keytabs/zookeeper-server.keytab"
>>
>> principal="zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM";
>>
>> };
>>
>>
>>
>> export
>> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaa
>> s_server.conf
>>
>>
>>
>> /conf/zookeeper_jaas_client.conf
>>
>>
>>
>> Client {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> storeKey=true
>>
>> useTicketCache=false
>>
>> keyTab="/etc/security/keytabs/zookeeper-client.keytab"
>>
>> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.C
>> OM";
>>
>> };
>>
>>
>>
>> export
>> CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaa
>> s_client.conf
>>
>>
>>
>> ·        And I’ve added this configuration to Accumulo:
>>
>>
>>
>> /usr/local/zookeeper/conf/zookeeper_jaas_client.conf
>>
>>
>>
>> Client {
>>
>> com.sun.security.auth.module.Krb5LoginModule required
>>
>> useKeyTab=true
>>
>> storeKey=true
>>
>> useTicketCache=false
>>
>> keyTab="/etc/security/keytabs/accumulo.keytab"
>>
>> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.C
>> OM";
>>
>> };
>>
>>
>>
>> export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf"
>>
>>
>>
>> This means that Zookeeper server starts by authenticating as principle zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM and Accumulo (a Zookeeper client) authenticates with Zookeeper using the zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM principle.
>>
>>
>>
>> When starting Accumulo for the first time and running through an initialisation, this authentication works fine and Accumulo starts as expected. But when you restart Accumulo, the master fails to start with this error:
>>
>>
>>
>> 2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor]
>> ERROR: Saw more than one ACL on the node
>>
>> 2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor]
>> ERROR: Expected
>> /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys
>> to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} ,
>> 31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=}
>>
>> ]
>>
>> 2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception,
>> exiting
>>
>> java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper is not protected.
>>
>>          at
>> org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDis
>> tributor.initialize(ZooAuthenticationKeyDistributor.java:86)
>>
>>          at org.apache.accumulo.master.Master.run(Master.java:1223)
>>
>>          at org.apache.accumulo.master.Master.main(Master.java:1434)
>>
>>          at
>> org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.j
>> ava:33)
>>
>>          at org.apache.accumulo.start.Main$1.run(Main.java:120)
>>
>>          at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> It looks like the information that Accumulo writes to Zookeeper during the initialisation has two ACLs associated to it – one for the zookeeper-client Kerberos principle (sasl) and one for the Accumulo secret (digest). These two ACLs seem to fail one of the master’s start-up checks and causes it to exit.
>>
>>
>>
>> Is there anyway of disabling the Accumulo secret so that I only have the Kerberos ACL? Or is there something wrong with the way I’ve tried to implement this that would cause this problem?
>>
>>
>>
>> I’d be very appreciative of any assistance.
>>
>>
>>
>> Many thanks,
>>
>>
>>
>> Oliver Jones

RE: Accumulo/Zookeeper Kerberos Integration

Posted by "Jones, Oliver" <ol...@cgi.com>.
Thank you for your response Christopher. I was unaware of that - I thought Accumulo could interacted with a kerberised Zookeeper. I will keep checking new releases to see if this changes.

Thanks, 

Ollie

-----Original Message-----
From: Christopher [mailto:ctubbsii@apache.org] 
Sent: 27 March 2019 02:12
To: accumulo-user <us...@accumulo.apache.org>
Subject: Re: Accumulo/Zookeeper Kerberos Integration

Accumulo does not currently support connecting to ZooKeeper using Kerberos as an alternative to the digest secret. Such a feature would need to be implemented.

On Tue, Mar 26, 2019 at 4:15 PM Jones, Oliver <ol...@cgi.com> wrote:
>
> Hello all,
>
>
>
> I sent this in a couple of months ago, but I’m not sure if I was subscribed to the mailing list correctly and didn’t see any replies – so I thought I’d try again.
>
>
>
> I’m having trouble getting Kerberos authentication to work between Zookeeper and Accumulo. I am not using any supporting platforms (e.g. Cloudera, Hortonworks) - this is all being done using Docker with single Accumulo (1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their own Docker network. My KDC is running on a separate CentOS machine, but can be reached by all of them. I have already managed to integrate Kerberos authentication with Hadoop and Accumulo, but cannot add Zookeeper client into the mix.
>
>
>
> ·        The Zookeeper container has this configuration:
>
>
>
> /conf/zoo.cfg
>
>
>
> # Kerberos Configuration
>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProv
> ider
>
> kerberos.removeHostFromPrincipal=true
>
> kerberos.removeRealmFromPrincipal=true
>
> requireClientAuthScheme=sasl
>
> jaasLoginRenew=3600000
>
>
>
> /conf/zookeeper_jaas_server.conf
>
>
>
> Server {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/zookeeper-server.keytab"
>
> principal="zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM";
>
> };
>
>
>
> export 
> SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaa
> s_server.conf
>
>
>
> /conf/zookeeper_jaas_client.conf
>
>
>
> Client {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/zookeeper-client.keytab"
>
> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.C
> OM";
>
> };
>
>
>
> export 
> CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaa
> s_client.conf
>
>
>
> ·        And I’ve added this configuration to Accumulo:
>
>
>
> /usr/local/zookeeper/conf/zookeeper_jaas_client.conf
>
>
>
> Client {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/accumulo.keytab"
>
> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.C
> OM";
>
> };
>
>
>
> export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf"
>
>
>
> This means that Zookeeper server starts by authenticating as principle zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM and Accumulo (a Zookeeper client) authenticates with Zookeeper using the zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM principle.
>
>
>
> When starting Accumulo for the first time and running through an initialisation, this authentication works fine and Accumulo starts as expected. But when you restart Accumulo, the master fails to start with this error:
>
>
>
> 2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor] 
> ERROR: Saw more than one ACL on the node
>
> 2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor] 
> ERROR: Expected 
> /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys 
> to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} , 
> 31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=}
>
> ]
>
> 2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception, 
> exiting
>
> java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper is not protected.
>
>         at 
> org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDis
> tributor.initialize(ZooAuthenticationKeyDistributor.java:86)
>
>         at org.apache.accumulo.master.Master.run(Master.java:1223)
>
>         at org.apache.accumulo.master.Master.main(Master.java:1434)
>
>         at 
> org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.j
> ava:33)
>
>         at org.apache.accumulo.start.Main$1.run(Main.java:120)
>
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> It looks like the information that Accumulo writes to Zookeeper during the initialisation has two ACLs associated to it – one for the zookeeper-client Kerberos principle (sasl) and one for the Accumulo secret (digest). These two ACLs seem to fail one of the master’s start-up checks and causes it to exit.
>
>
>
> Is there anyway of disabling the Accumulo secret so that I only have the Kerberos ACL? Or is there something wrong with the way I’ve tried to implement this that would cause this problem?
>
>
>
> I’d be very appreciative of any assistance.
>
>
>
> Many thanks,
>
>
>
> Oliver Jones

Re: Accumulo/Zookeeper Kerberos Integration

Posted by Christopher <ct...@apache.org>.
Accumulo does not currently support connecting to ZooKeeper using
Kerberos as an alternative to the digest secret. Such a feature would
need to be implemented.

On Tue, Mar 26, 2019 at 4:15 PM Jones, Oliver <ol...@cgi.com> wrote:
>
> Hello all,
>
>
>
> I sent this in a couple of months ago, but I’m not sure if I was subscribed to the mailing list correctly and didn’t see any replies – so I thought I’d try again.
>
>
>
> I’m having trouble getting Kerberos authentication to work between Zookeeper and Accumulo. I am not using any supporting platforms (e.g. Cloudera, Hortonworks) - this is all being done using Docker with single Accumulo (1.8.1), Zookeeper (3.4.10) and Hadoop (2.8.2) containers running within their own Docker network. My KDC is running on a separate CentOS machine, but can be reached by all of them. I have already managed to integrate Kerberos authentication with Hadoop and Accumulo, but cannot add Zookeeper client into the mix.
>
>
>
> ·        The Zookeeper container has this configuration:
>
>
>
> /conf/zoo.cfg
>
>
>
> # Kerberos Configuration
>
> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
>
> kerberos.removeHostFromPrincipal=true
>
> kerberos.removeRealmFromPrincipal=true
>
> requireClientAuthScheme=sasl
>
> jaasLoginRenew=3600000
>
>
>
> /conf/zookeeper_jaas_server.conf
>
>
>
> Server {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/zookeeper-server.keytab"
>
> principal="zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM";
>
> };
>
>
>
> export SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_server.conf
>
>
>
> /conf/zookeeper_jaas_client.conf
>
>
>
> Client {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/zookeeper-client.keytab"
>
> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM";
>
> };
>
>
>
> export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/zookeeper_jaas_client.conf
>
>
>
> ·        And I’ve added this configuration to Accumulo:
>
>
>
> /usr/local/zookeeper/conf/zookeeper_jaas_client.conf
>
>
>
> Client {
>
> com.sun.security.auth.module.Krb5LoginModule required
>
> useKeyTab=true
>
> storeKey=true
>
> useTicketCache=false
>
> keyTab="/etc/security/keytabs/accumulo.keytab"
>
> principal="zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM";
>
> };
>
>
>
> export ACCUMULO_JAAS_CONF="$ZOOKEEPER_HOME/conf/zookeeper_jaas_client.conf"
>
>
>
> This means that Zookeeper server starts by authenticating as principle zookeeper/zk-kerberos.accumulo-krb-network@EXAMPLE.COM and Accumulo (a Zookeeper client) authenticates with Zookeeper using the zookeeper-client/zk-kerberos.accumulo-krb-network@EXAMPLE.COM principle.
>
>
>
> When starting Accumulo for the first time and running through an initialisation, this authentication works fine and Accumulo starts as expected. But when you restart Accumulo, the master fails to start with this error:
>
>
>
> 2019-02-07 10:52:42,856 [delegation.ZooAuthenticationKeyDistributor] ERROR: Saw more than one ACL on the node
>
> 2019-02-07 10:52:42,858 [delegation.ZooAuthenticationKeyDistributor] ERROR: Expected /accumulo/ee2ddad6-9df3-43a0-84d4-e9713ae9058c/delegation_token_keys to have ACLs [31,s{'auth,'} ] but was [31,s{'sasl,'zookeeper-client} , 31,s{'digest,'accumulo:diZNqb4D71cy0fGxC3meE2ZYWyE=}
>
> ]
>
> 2019-02-07 10:52:42,858 [master.Master] ERROR: Unexpected exception, exiting
>
> java.lang.IllegalStateException: Delegation token secret key node in ZooKeeper is not protected.
>
>         at org.apache.accumulo.server.security.delegation.ZooAuthenticationKeyDistributor.initialize(ZooAuthenticationKeyDistributor.java:86)
>
>         at org.apache.accumulo.master.Master.run(Master.java:1223)
>
>         at org.apache.accumulo.master.Master.main(Master.java:1434)
>
>         at org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.java:33)
>
>         at org.apache.accumulo.start.Main$1.run(Main.java:120)
>
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> It looks like the information that Accumulo writes to Zookeeper during the initialisation has two ACLs associated to it – one for the zookeeper-client Kerberos principle (sasl) and one for the Accumulo secret (digest). These two ACLs seem to fail one of the master’s start-up checks and causes it to exit.
>
>
>
> Is there anyway of disabling the Accumulo secret so that I only have the Kerberos ACL? Or is there something wrong with the way I’ve tried to implement this that would cause this problem?
>
>
>
> I’d be very appreciative of any assistance.
>
>
>
> Many thanks,
>
>
>
> Oliver Jones