Proposal for an End-to-End encryption in Kafka

[Hendrik Dev <he...@gmail.com>]

Proposal for an End-to-End encryption in Kafka:

https://github.com/salyh/kafka/commit/151a0340645f4165cb120f79df544326e5a0db6f

Introduce EncryptingSerializer and DecryptingDeserializer.
They act as wrappers for inner (de)serializer and provide transparent
AES and RSA based end-to-end encryption.
So all messages stored in Kafka which passed the EncryptingSerializer
are stored in an encrypted manner.
The DecryptingDeserializer can detect if a message is encrypted and if
not its just passed through as it is.

Here is how it works:
https://github.com/salyh/kafka/blob/151a0340645f4165cb120f79df544326e5a0db6f/clients/src/main/java/org/apache/kafka/common/serialization/EncrpytingSerializer.java

Currently the code is not threadsafe and not optimized or benchmarked.
Its just a proof of concept to start a discussion if such a
contribution would be useful.
Depending on the RSA keysize a overhead of additional 41+(RSA keysize
in bytes) bytes per message are added and for each message produced
(or consumed) a AES en-/decryption needs to be performed. RSA
encryption of the AES key happens not very often cause this could be
cached - so RSA en-/decryption is likely not a big issue regarding
performance.

Note: AES encryption on modern hardware with a recent Java runtime is
typically hardware accelerated (by leveraging AES-NI and -XX:+UseAES
-XX:+UseAESIntrinsics)

Feedback and comments appreciated.

Thanks
Hendrik

-- 
Hendrik Saly (salyh, hendrikdev22)
@hendrikdev22
PGP: 0x22D7F6EC
salyh <AT> apache <DOT> org