You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sönke Liebau <so...@opencore.com.INVALID> on 2018/10/02 11:19:36 UTC
Discussion on requirements for Data Encryption functionality in Kafka (KIP-317)
Hi all,
I have created KIP-317 [1] a while ago, which outlines an implementation
proposal to add transparent data encryption functionality to Kafka. The KIP
in its current form is somewhat rigid in its implementation, I will rework
this to become extensible over the next few days to allow for additional
implementations.
I have discussed the current method of providing keys with a colleague and
while we agree that this is a valid use case for some people, there are
certainly a lot of other valid use cases out there as well.
To ensure that the initial implementation provides the necessary
flexibility I'd like some feedback from the community on what requirements
they would have around data encryption and key management.
The following questions should serve as a starting point for the
discussion, please feel free to address anything that comes to mind which I
have not mentioned here:
- Should encryption be configurable rather on the client or on the broker
and be pushed down to the client?
- Where should keys be stored?
- How much flexibility around keys is necessary - is there for example a
use case that would decide on a per message basis which key to use?
(imagine a topic containing top secret, secret and public data with three
different keys)
- Do we need functionality to prohibit publishing unencrypted messages to
topics based on that topics setup?
Of course the mailing list is the first place that discussions like these
should take place, but sometimes I find a face to face discussion can be
quite useful as well, especially when discussing non-trivial topics (like
encryption). I have reached out to the organizers of the upcoming Kafka
Summit in SF and there might be a chance for us to get a room with a
whiteboard at some point (probably during lunch, when the room is otherwise
unused). Would people be interested in meeting up for 20 minutes to discuss
this in person? I'd be happy to provide a summary on the mailing list
afterwards of course.
Look forward to hearing from all of you!
Best regards,
Sönke
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+transparent+data+encryption+functionality
Re: Discussion on requirements for Data Encryption functionality in
Kafka (KIP-317)
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Mike,
that sounds good! I've not yet received any other feedback, but worst case
scenario is that just the two of us discuss this over a cup of coffee :)
I'll talk to the Summit organizers again about some sort of venue and get
back to you once I know more. Maybe we can get a few more people to join
the discussion once some more details are known.
Other than that, I am also very happy to gather feedback on the mailing
list from people who won't be able to make it to the Summit. So anybody who
can come up with some thoughts or requirements around encryption
functionality for Kafka, please don't hesitate to chime in!
Best regards,
Sönke
On Wed, Oct 3, 2018 at 3:03 AM mikegray831@gmail.com <mi...@gmail.com>
wrote:
> Hi Sönke,
>
> I would be very interested in participating in this conversation. Very
> interested in how TDE might work in Kafka! I’m coming with several
> colleagues and will see if they’re interested in participating as well.
>
> Thanks,
> Mike Grayson
>
> On 2018/10/02 11:19:36, Sönke Liebau <so...@opencore.com.INVALID>
> wrote:
> > Hi all,
> >
> > I have created KIP-317 [1] a while ago, which outlines an implementation
> > proposal to add transparent data encryption functionality to Kafka. The
> KIP
> > in its current form is somewhat rigid in its implementation, I will
> rework
> > this to become extensible over the next few days to allow for additional
> > implementations.
> >
> > I have discussed the current method of providing keys with a colleague
> and
> > while we agree that this is a valid use case for some people, there are
> > certainly a lot of other valid use cases out there as well.
> > To ensure that the initial implementation provides the necessary
> > flexibility I'd like some feedback from the community on what
> requirements
> > they would have around data encryption and key management.
> >
> > The following questions should serve as a starting point for the
> > discussion, please feel free to address anything that comes to mind
> which I
> > have not mentioned here:
> >
> > - Should encryption be configurable rather on the client or on the broker
> > and be pushed down to the client?
> > - Where should keys be stored?
> > - How much flexibility around keys is necessary - is there for example a
> > use case that would decide on a per message basis which key to use?
> > (imagine a topic containing top secret, secret and public data with three
> > different keys)
> > - Do we need functionality to prohibit publishing unencrypted messages to
> > topics based on that topics setup?
> >
> > Of course the mailing list is the first place that discussions like these
> > should take place, but sometimes I find a face to face discussion can be
> > quite useful as well, especially when discussing non-trivial topics (like
> > encryption). I have reached out to the organizers of the upcoming Kafka
> > Summit in SF and there might be a chance for us to get a room with a
> > whiteboard at some point (probably during lunch, when the room is
> otherwise
> > unused). Would people be interested in meeting up for 20 minutes to
> discuss
> > this in person? I'd be happy to provide a summary on the mailing list
> > afterwards of course.
> >
> > Look forward to hearing from all of you!
> >
> > Best regards,
> > Sönke
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+transparent+data+encryption+functionality
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Discussion on requirements for Data Encryption functionality in Kafka (KIP-317)
Posted by mi...@gmail.com,
mi...@gmail.com.
Hi Sönke,
I would be very interested in participating in this conversation. Very interested in how TDE might work in Kafka! I’m coming with several colleagues and will see if they’re interested in participating as well.
Thanks,
Mike Grayson
On 2018/10/02 11:19:36, Sönke Liebau <so...@opencore.com.INVALID> wrote:
> Hi all,
>
> I have created KIP-317 [1] a while ago, which outlines an implementation
> proposal to add transparent data encryption functionality to Kafka. The KIP
> in its current form is somewhat rigid in its implementation, I will rework
> this to become extensible over the next few days to allow for additional
> implementations.
>
> I have discussed the current method of providing keys with a colleague and
> while we agree that this is a valid use case for some people, there are
> certainly a lot of other valid use cases out there as well.
> To ensure that the initial implementation provides the necessary
> flexibility I'd like some feedback from the community on what requirements
> they would have around data encryption and key management.
>
> The following questions should serve as a starting point for the
> discussion, please feel free to address anything that comes to mind which I
> have not mentioned here:
>
> - Should encryption be configurable rather on the client or on the broker
> and be pushed down to the client?
> - Where should keys be stored?
> - How much flexibility around keys is necessary - is there for example a
> use case that would decide on a per message basis which key to use?
> (imagine a topic containing top secret, secret and public data with three
> different keys)
> - Do we need functionality to prohibit publishing unencrypted messages to
> topics based on that topics setup?
>
> Of course the mailing list is the first place that discussions like these
> should take place, but sometimes I find a face to face discussion can be
> quite useful as well, especially when discussing non-trivial topics (like
> encryption). I have reached out to the organizers of the upcoming Kafka
> Summit in SF and there might be a chance for us to get a room with a
> whiteboard at some point (probably during lunch, when the room is otherwise
> unused). Would people be interested in meeting up for 20 minutes to discuss
> this in person? I'd be happy to provide a summary on the mailing list
> afterwards of course.
>
> Look forward to hearing from all of you!
>
> Best regards,
> Sönke
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+transparent+data+encryption+functionality
>