You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ismael Juma <is...@juma.me.uk> on 2017/09/04 07:44:18 UTC
Re: Plans to improve SSL performance in Kafka, for 0.10.x?
By the way, in-kernel TLS has now landed in the Linux kernel:
https://github.com/torvalds/linux/blob/master/Documentation/networking/tls.txt
There is work in progress to take advantage of that in OpenSSL:
https://github.com/Mellanox/tls-openssl
Ismael
On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:
> Yeah, that's why I mentioned it with a caveat :) Someone (I can't recall
> who, but it was someone I consider reasonably knowledgable as I actually
> gave it some weight) mentioned it, but I haven't looked into it further
> than that. I agree that I don't see how this is going to help us at the app
> layer.
>
> -Todd
>
> On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Todd,
> >
> > Thanks for sharing your experience enabling TLS in your clusters. Very
> > helpful. One comment below.
> >
> > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> > >
> > > Right now, we're specifically avoiding moving consume traffic to SSL,
> due
> > > to the zero copy send issue. Now I've been told (but I have not
> > > investigated) that OpenSSL can solve this. It would probably be a good
> > use
> > > of time to look into that further.
> > >
> >
> > As far as I know, OpenSSL can reduce the TLS overhead, but we will still
> > lose the zero-copy optimisation. There is some attempts at making it
> > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > probably too early for us to consider that for Kafka.
> >
> > Ismael
> >
> > [1] https://lwn.net/Articles/666509/
> > [2]
> > http://techblog.netflix.com/2016/08/protecting-netflix-
> > viewing-privacy-at.html
> >
>
>
> --
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>
Re: Plans to improve SSL performance in Kafka, for 0.10.x?
Posted by Ismael Juma <is...@juma.me.uk>.
Great!
Ismael
On Mon, Sep 4, 2017 at 1:43 PM, Todd Palino <tp...@gmail.com> wrote:
> I will say that we've been turning on TLS consume lately (including using
> it for IBP, which we've been doing for a while) and we haven't seen any of
> the performance concerns that we originally did. Very little hit at all.
>
> -Todd
>
> On Sep 4, 2017 12:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
>
> > By the way, in-kernel TLS has now landed in the Linux kernel:
> >
> > https://github.com/torvalds/linux/blob/master/
> > Documentation/networking/tls.txt
> >
> > There is work in progress to take advantage of that in OpenSSL:
> >
> > https://github.com/Mellanox/tls-openssl
> >
> > Ismael
> >
> >
> > On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > Yeah, that's why I mentioned it with a caveat :) Someone (I can't
> recall
> > > who, but it was someone I consider reasonably knowledgable as I
> actually
> > > gave it some weight) mentioned it, but I haven't looked into it further
> > > than that. I agree that I don't see how this is going to help us at the
> > app
> > > layer.
> > >
> > > -Todd
> > >
> > > On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Todd,
> > > >
> > > > Thanks for sharing your experience enabling TLS in your clusters.
> Very
> > > > helpful. One comment below.
> > > >
> > > > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > > > <javascript:;>> wrote:
> > > > >
> > > > > Right now, we're specifically avoiding moving consume traffic to
> SSL,
> > > due
> > > > > to the zero copy send issue. Now I've been told (but I have not
> > > > > investigated) that OpenSSL can solve this. It would probably be a
> > good
> > > > use
> > > > > of time to look into that further.
> > > > >
> > > >
> > > > As far as I know, OpenSSL can reduce the TLS overhead, but we will
> > still
> > > > lose the zero-copy optimisation. There is some attempts at making it
> > > > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > > > probably too early for us to consider that for Kafka.
> > > >
> > > > Ismael
> > > >
> > > > [1] https://lwn.net/Articles/666509/
> > > > [2]
> > > > http://techblog.netflix.com/2016/08/protecting-netflix-
> > > > viewing-privacy-at.html
> > > >
> > >
> > >
> > > --
> > > *Todd Palino*
> > > Staff Site Reliability Engineer
> > > Data Infrastructure Streaming
> > >
> > >
> > >
> > > linkedin.com/in/toddpalino
> > >
> >
>
Re: Plans to improve SSL performance in Kafka, for 0.10.x?
Posted by Todd Palino <tp...@gmail.com>.
I will say that we've been turning on TLS consume lately (including using
it for IBP, which we've been doing for a while) and we haven't seen any of
the performance concerns that we originally did. Very little hit at all.
-Todd
On Sep 4, 2017 12:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> By the way, in-kernel TLS has now landed in the Linux kernel:
>
> https://github.com/torvalds/linux/blob/master/
> Documentation/networking/tls.txt
>
> There is work in progress to take advantage of that in OpenSSL:
>
> https://github.com/Mellanox/tls-openssl
>
> Ismael
>
>
> On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > Yeah, that's why I mentioned it with a caveat :) Someone (I can't recall
> > who, but it was someone I consider reasonably knowledgable as I actually
> > gave it some weight) mentioned it, but I haven't looked into it further
> > than that. I agree that I don't see how this is going to help us at the
> app
> > layer.
> >
> > -Todd
> >
> > On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Todd,
> > >
> > > Thanks for sharing your experience enabling TLS in your clusters. Very
> > > helpful. One comment below.
> > >
> > > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > > <javascript:;>> wrote:
> > > >
> > > > Right now, we're specifically avoiding moving consume traffic to SSL,
> > due
> > > > to the zero copy send issue. Now I've been told (but I have not
> > > > investigated) that OpenSSL can solve this. It would probably be a
> good
> > > use
> > > > of time to look into that further.
> > > >
> > >
> > > As far as I know, OpenSSL can reduce the TLS overhead, but we will
> still
> > > lose the zero-copy optimisation. There is some attempts at making it
> > > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > > probably too early for us to consider that for Kafka.
> > >
> > > Ismael
> > >
> > > [1] https://lwn.net/Articles/666509/
> > > [2]
> > > http://techblog.netflix.com/2016/08/protecting-netflix-
> > > viewing-privacy-at.html
> > >
> >
> >
> > --
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>