customising security across the whole of Confluent platform

[Joris Peeters <jo...@gmail.com>]

Hello,

I am trying to come up with a good security approach for a Kafka project
inside our company. I see there's a variety of options available
(ACL/RBAC/certificates/...), and I'm hoping someone can suggest a few
possibilities.
Some points of interest,

- Kafka will be used both by end-users/researchers (inside the company) and
by some microservices. The users are on a wide variety of platforms
(Windows/Linux, JVM/C++/Python/R/Javascript/MATLAB/...). The microservices
are probably JVM/Python.
- We'd like to make heavy use of both the Schema Registry and REST proxy,
for all users,
- Users & groups are in LDAP,
- Permissioning would need to be granular and at the level of Kafka
(topics, consumer groups, ...), the Schema Registry, the REST Proxy, ... -
based on LDAP group membership.
- Ideally we'd like to use user/password authentication throughout, which
is the most widely supported mechanism (as we use many different
platforms). The user is the LDAP username, but the password is an API key
that users manage separately. For security reasons we cannot use the LDAP
password, as it would be too likely to leak (e.g. in git etc). (We already
use the LDAP user + API key combo for authentication in other places).

For Kafka, I've played around with an approach similar to
https://github.com/navikt/kafka-plain-saslserver-2-ad, i.e. writing classes
that extend
org.apache.kafka.common.security.auth.AuthenticateCallbackHandler and
kafka.security.auth.Authorizer for providing our own rules, which works
perfectly fine. In this case,
- the authenticator checks the username against our own database of (user
-> [apiKeys]),
- the authoriser meaningfully overrides only the "authorize(..)" function
(leaving all the *Acl ones as no-ops) and leverages LDAP (in particular
group membership) to decide upon authorization, using an additional
database encoding group permissions.
This is a bit hacky - and more a proof-of-concept than anything else, but
it does seem to cover our needs, as far as Kafka goes.

However, I believe this is only available for Kafka itself, and a similar
approach wouldn't extend to the REST proxy and the Schema Registry (and
Connectors and the Control Center). Ideally, we want a centralized auth
service.
In that context, I've been looking at Role-Based Access Control. I think it
would suit the majority of our purposes, except for the fact that it seems
to authenticate using LDAP only (using LDAP user/pass for authentication),
which we cannot do. I was wondering,

- Is it possible to extend the RBAC with a custom authenticator, that maps
(user+passw) -> bool? We could probably still use LDAP directly for
authorisation. If not, though - is a similar class extension possible for
authorisation as well? If I understood correctly, this would need to be
done inside the Metadata Service.
- An alternative could be to run a separate LDAP server, identical to our
company-central one, but where the user passwords are set to their API
keys. That would be quite a faff, though.

If this is not possible - what is it that makes our use case different from
the norm? Is it that,
- generally people are OK with passing LDAP passwords?
- generally "external" users don't need the schema registry or REST proxy
(i.e. they are only used by internal microservices), so they can be
firewalled off?
- generally all enterprise consumers are on the JVM, making Kerberos auth
more plausible than in our case?
- ... ?

Note that I do not have a ton of experience with securing Kafka, so the
above may well contain numerous mistaken/out-of-date assumptions. In any
case, thanks for any suggestions.

Best,
-Joris.

Re: customising security across the whole of Confluent platform

[Rajini Sivaram <ra...@gmail.com>]

Hi Joris,

I have forwarded your mail to *security@confluent.io
<se...@confluent.io>* since it is about security in the Confluent
Platform rather than in Apache Kafka.

Regards,

Rajini

On Tue, Sep 17, 2019 at 11:35 AM Joris Peeters <jo...@gmail.com>
wrote:

> Hello,
>
> I am trying to come up with a good security approach for a Kafka project
> inside our company. I see there's a variety of options available
> (ACL/RBAC/certificates/...), and I'm hoping someone can suggest a few
> possibilities.
> Some points of interest,
>
> - Kafka will be used both by end-users/researchers (inside the company) and
> by some microservices. The users are on a wide variety of platforms
> (Windows/Linux, JVM/C++/Python/R/Javascript/MATLAB/...). The microservices
> are probably JVM/Python.
> - We'd like to make heavy use of both the Schema Registry and REST proxy,
> for all users,
> - Users & groups are in LDAP,
> - Permissioning would need to be granular and at the level of Kafka
> (topics, consumer groups, ...), the Schema Registry, the REST Proxy, ... -
> based on LDAP group membership.
> - Ideally we'd like to use user/password authentication throughout, which
> is the most widely supported mechanism (as we use many different
> platforms). The user is the LDAP username, but the password is an API key
> that users manage separately. For security reasons we cannot use the LDAP
> password, as it would be too likely to leak (e.g. in git etc). (We already
> use the LDAP user + API key combo for authentication in other places).
>
> For Kafka, I've played around with an approach similar to
> https://github.com/navikt/kafka-plain-saslserver-2-ad, i.e. writing
> classes
> that extend
> org.apache.kafka.common.security.auth.AuthenticateCallbackHandler and
> kafka.security.auth.Authorizer for providing our own rules, which works
> perfectly fine. In this case,
> - the authenticator checks the username against our own database of (user
> -> [apiKeys]),
> - the authoriser meaningfully overrides only the "authorize(..)" function
> (leaving all the *Acl ones as no-ops) and leverages LDAP (in particular
> group membership) to decide upon authorization, using an additional
> database encoding group permissions.
> This is a bit hacky - and more a proof-of-concept than anything else, but
> it does seem to cover our needs, as far as Kafka goes.
>
> However, I believe this is only available for Kafka itself, and a similar
> approach wouldn't extend to the REST proxy and the Schema Registry (and
> Connectors and the Control Center). Ideally, we want a centralized auth
> service.
> In that context, I've been looking at Role-Based Access Control. I think it
> would suit the majority of our purposes, except for the fact that it seems
> to authenticate using LDAP only (using LDAP user/pass for authentication),
> which we cannot do. I was wondering,
>
> - Is it possible to extend the RBAC with a custom authenticator, that maps
> (user+passw) -> bool? We could probably still use LDAP directly for
> authorisation. If not, though - is a similar class extension possible for
> authorisation as well? If I understood correctly, this would need to be
> done inside the Metadata Service.
> - An alternative could be to run a separate LDAP server, identical to our
> company-central one, but where the user passwords are set to their API
> keys. That would be quite a faff, though.
>
> If this is not possible - what is it that makes our use case different from
> the norm? Is it that,
> - generally people are OK with passing LDAP passwords?
> - generally "external" users don't need the schema registry or REST proxy
> (i.e. they are only used by internal microservices), so they can be
> firewalled off?
> - generally all enterprise consumers are on the JVM, making Kerberos auth
> more plausible than in our case?
> - ... ?
>
> Note that I do not have a ton of experience with securing Kafka, so the
> above may well contain numerous mistaken/out-of-date assumptions. In any
> case, thanks for any suggestions.
>
> Best,
> -Joris.
>