Using Kerberos auth *with keytab updates* with Storm?

[Steve Miller <st...@idrathernotsay.com>]

Hi.  I'm trying to set up a Storm cluster using Kerberos authentication.  I can make that work in a static case: that is, I can generate keytabs and jaas.conf files and the right stuff in storm.yaml and zoo.cfg and the like and it's all working.

Now, for my next trick: being able to change the keytabs on a routine basis, without having to take the whole cluster down.

There are two things in particular about which I'm unclear.

First, and most importantly, the examples I've found for setting this up all use a single Kerberos principal (storm@REALM) to authenticate to Zookeeper.  And of course something sets the ACL on /storm and the stuff below it to be storm@REALM.

So if we update the credentials for storm@REALM, we presumably need to update those credentials on every Storm host in fairly short order, or individual supervisors (or Nimbus or the UI) won't be able to authenticate to Zookeeper.  But coordinating that at more-or-less the same time seems tricky, especially in an environment where some central host can't just reach out and coordinate all the changes at once.

I don't know if it's possible to set storm.zookeeper.superACL to a list of credentials, one per host, so that everything can still make changes under /storm but all the individual hosts can update their credentials on their own schedule.

Second, I'm not quite sure what happens to roll the keytabs themselves.  If I update a keytab, do I have to restart the supervisor (or whatever) that's using that keytab?  Or do the various components just try to authenticate periodically, and if they fail, re-read the keytab file -- or do they never cache the contents of the keytab file, and always read it fresh, so if we just update the keytab file with the latest key shortly after we change the credential on the KDC?

If someone could share some details on how they handle key rolls on a production cluster, that'd be hugely useful.  I'd be happy to try to update the "running an authenticated Storm cluster" docs to reflect how that sort of detail works.

And please accept my apologies in advance if I'm saying something dumb here: I am far from being an expert on Kerberos.

Thanks!

    -Steve

Re: Using Kerberos auth *with keytab updates* with Storm?

[Steve Miller <st...@idrathernotsay.com>]

BTW I did find the source for where storm.zookeeper.suoerACL is parsed, at least for Storm 1.1.0 and 1.1.1, and it only supports a string of the form "sasl:thing" -- it knows more than is really healthy for it about the format there.

It seems like if it expected a YAML sequence (or array or whatever they're called in YAML-land) it'd be possible to clean up the code a bit in getWorkerACL() in storm-core/src/jvm/org/apache/storm/utils/Utils.java, and add support for setting this to cover multiple credentials if that's what the administrator wants to do.

Still, if someone who is using Storm with Kerberos authentication could explain to me what they're doing when they have to update keys, that'd be massively useful.

    -Steve

> On Sep 7, 2017, at 11:45 AM, Steve Miller <st...@idrathernotsay.com> wrote:
> 
> Hi.  I'm trying to set up a Storm cluster using Kerberos authentication.  I can make that work in a static case: that is, I can generate keytabs and jaas.conf files and the right stuff in storm.yaml and zoo.cfg and the like and it's all working.
> 
> Now, for my next trick: being able to change the keytabs on a routine basis, without having to take the whole cluster down.
> 
> There are two things in particular about which I'm unclear.
> 
> First, and most importantly, the examples I've found for setting this up all use a single Kerberos principal (storm@REALM) to authenticate to Zookeeper.  And of course something sets the ACL on /storm and the stuff below it to be storm@REALM.
> 
> So if we update the credentials for storm@REALM, we presumably need to update those credentials on every Storm host in fairly short order, or individual supervisors (or Nimbus or the UI) won't be able to authenticate to Zookeeper.  But coordinating that at more-or-less the same time seems tricky, especially in an environment where some central host can't just reach out and coordinate all the changes at once.
> 
> I don't know if it's possible to set storm.zookeeper.superACL to a list of credentials, one per host, so that everything can still make changes under /storm but all the individual hosts can update their credentials on their own schedule.
> 
> Second, I'm not quite sure what happens to roll the keytabs themselves.  If I update a keytab, do I have to restart the supervisor (or whatever) that's using that keytab?  Or do the various components just try to authenticate periodically, and if they fail, re-read the keytab file -- or do they never cache the contents of the keytab file, and always read it fresh, so if we just update the keytab file with the latest key shortly after we change the credential on the KDC?
> 
> If someone could share some details on how they handle key rolls on a production cluster, that'd be hugely useful.  I'd be happy to try to update the "running an authenticated Storm cluster" docs to reflect how that sort of detail works.
> 
> And please accept my apologies in advance if I'm saying something dumb here: I am far from being an expert on Kerberos.
> 
> Thanks!
> 
>    -Steve
> 
> 
>