Plans to improve SSL performance in Kafka, for 0.10.x?

[Jaikiran Pai <ja...@gmail.com>]

We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently 
decided to start using SSL for Kafka communication between broker and 
clients. Right now, we have a pretty basic setup with just 1 broker with 
SSL keystore setup and the Java client(s) communicate using the 
Producer/Consumer APIs against this single broker. There's no client 
auth (intentionally) right now. We also have plain text enabled for the 
initial testing.

What we have noticed is that the consumer/producer performance when SSL 
is enabled is noticeably poor when compared to plain text. I understand 
that there are expected to be performance impacts when SSL is enabled 
but the order of magnitude is too high and in fact it shows up in a very 
noticeable fashion in our product. I do have the numbers, but I haven't 
been able to narrow it down yet (because there's nothing that stands 
out, except that it's slow). Our application code is exactly the same 
between non-SSL and SSL usage.

Furthermore, I'm aware of this specific JIRA in Kafka 
https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a 
similar issue. So what I would like to know is, in context of Kafka 
0.10.x releases and Java 8 support, are there any timelines that the dev 
team is looking for in terms of improving this performance issue (which 
I believe requires usage of OpenSSL or other non-JDK implementations of 
SSLEngine)? We would like to go for GA of our product in the next couple 
of months and in order to do that, we do plan to have Kafka over SSL 
working with reasonably good performance, but the current performance 
isn't promising. Expecting this to be fixed in the next couple of months 
and have it available in 0.10.x is probably too much to expect, but if 
we know the plans around this, we should be able to come up with a plan 
of our own for our product.


-Jaikiran

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Tom Crayford <tc...@heroku.com>]

On Tue, Sep 6, 2016 at 12:07 PM, Ismael Juma <is...@juma.me.uk> wrote:

> On Tue, Sep 6, 2016 at 11:35 AM, Tom Crayford <tc...@heroku.com>
> wrote:
>
> > Our performance tests were run with the JVM producer and consumer, and we
> > didn't notice any real slowdown on that side either when enabling SSL.
> Just
> > to check the obvious: are your producers and consumers going over a fast
> > connection to the broker?
> >
>
> This is interesting because we did notice a slowdown in our performance
> tests while developing the feature. The conclusion was that it was still
> possible to saturate the network with the right configuration, but your
> results seem much better. Can you share any specifics of the performance
> tests? Also, are the tests network bound?
>

Yes, entirely network bound, with 100 byte messages. We're not doing
anything particularly interesting for either the brokers or the clients.


>
> Ismael
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Ismael Juma <is...@juma.me.uk>]

On Tue, Sep 6, 2016 at 11:35 AM, Tom Crayford <tc...@heroku.com> wrote:

> Our performance tests were run with the JVM producer and consumer, and we
> didn't notice any real slowdown on that side either when enabling SSL. Just
> to check the obvious: are your producers and consumers going over a fast
> connection to the broker?
>

This is interesting because we did notice a slowdown in our performance
tests while developing the feature. The conclusion was that it was still
possible to saturate the network with the right configuration, but your
results seem much better. Can you share any specifics of the performance
tests? Also, are the tests network bound?

Ismael

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Tom Crayford <tc...@heroku.com>]

> We have a single broker

Kafka isn't really designed to run with a single broker setup. Whilst I
doubt it impacts your problems here, you should immediately bump your
broker count to 3 or more.

Our performance tests were run with the JVM producer and consumer, and we
didn't notice any real slowdown on that side either when enabling SSL. Just
to check the obvious: are your producers and consumers going over a fast
connection to the broker?

> This GC you notice, is it on the broker(s) or on the clients which hosts
the producer/consumers? Either way, this is a good hint and something I
hadn't considered in my setup. I'll see if I can get some GC logs/metrics
on our setup for this. Thanks for the input.

On the broker - we never looked at consumers or producers, but that might
be a good place to look.

On Tue, Sep 6, 2016 at 5:40 AM, Jaikiran Pai <ja...@gmail.com>
wrote:

> On Tuesday 06 September 2016 09:42 AM, Jaikiran Pai wrote:
>
>> Hi Todd,
>>
>> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
>>> SHA1PRNG.
>>>
>>
>> Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our
>> setup. However, right now, (intentionally) we have single broker within the
>> setup and based on the JIRA details, I believe this one won't affect us for
>> now.
>>
>
> I read that JIRA again and saw the stacktrace. I might be wrong when I
> said this probably doesn't affect us given that the code path appears to
> the same for other SSL interactions. I'm going to introduce this patch (or
> in fact the latest 0.10 branch) in our setup and see how it goes.
>
> -Jaikiran
>
>
>
>
>>
>>
>>> Right now, we're specifically avoiding moving consume traffic to SSL, due
>>> to the zero copy send issue. Now I've been told (but I have not
>>> investigated) that OpenSSL can solve this. It would probably be a good
>>> use
>>> of time to look into that further.
>>>
>>
>> Unfortunately, the way our application is written, we don't have the
>> ability to enable SSL for producers and disable it for consumers. Changing
>> the application to support something like that is doable, but I will have
>> to make sure that this is one of the reasons for the slowness.
>>
>>
>> Thanks for all your inputs. I was trying to avoid coming up with a
>> reproducer which closely matches our application usage scenario (given our
>> tight delivery timelines for the product), but from what you say and a
>> reply Tom Crayford in this thread, I am starting to feel that not everyone
>> is affected to the extent we are. I will see if I can narrow to this down
>> to something more concrete.
>>
>> -Jaikiran
>>
>>
>>
>>
>>
>>> That said, switching the message format to the newer option (KIP-31 I
>>> believe?) will result in the brokers not needing to recompress message
>>> batches that are produced. This should result in a significant reduction
>>> in
>>> CPU usage, which may offset the cost of SSL. We haven't had a chance to
>>> fully investigate this, however, as changing that config depends on the
>>> clients being updated to support the new format.
>>>
>>> -Todd
>>>
>>> On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com>
>>> wrote:
>>>
>>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
>>>> decided to start using SSL for Kafka communication between broker and
>>>> clients. Right now, we have a pretty basic setup with just 1 broker with
>>>> SSL keystore setup and the Java client(s) communicate using the
>>>> Producer/Consumer APIs against this single broker. There's no client
>>>> auth
>>>> (intentionally) right now. We also have plain text enabled for the
>>>> initial
>>>> testing.
>>>>
>>>> What we have noticed is that the consumer/producer performance when SSL
>>>> is
>>>> enabled is noticeably poor when compared to plain text. I understand
>>>> that
>>>> there are expected to be performance impacts when SSL is enabled but the
>>>> order of magnitude is too high and in fact it shows up in a very
>>>> noticeable
>>>> fashion in our product. I do have the numbers, but I haven't been able
>>>> to
>>>> narrow it down yet (because there's nothing that stands out, except that
>>>> it's slow). Our application code is exactly the same between non-SSL and
>>>> SSL usage.
>>>>
>>>> Furthermore, I'm aware of this specific JIRA in Kafka
>>>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
>>>> similar issue. So what I would like to know is, in context of Kafka
>>>> 0.10.x
>>>> releases and Java 8 support, are there any timelines that the dev team
>>>> is
>>>> looking for in terms of improving this performance issue (which I
>>>> believe
>>>> requires usage of OpenSSL or other non-JDK implementations of
>>>> SSLEngine)?
>>>> We would like to go for GA of our product in the next couple of months
>>>> and
>>>> in order to do that, we do plan to have Kafka over SSL working with
>>>> reasonably good performance, but the current performance isn't
>>>> promising.
>>>> Expecting this to be fixed in the next couple of months and have it
>>>> available in 0.10.x is probably too much to expect, but if we know the
>>>> plans around this, we should be able to come up with a plan of our own
>>>> for
>>>> our product.
>>>>
>>>>
>>>> -Jaikiran
>>>>
>>>>
>>>
>>
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Jaikiran Pai <ja...@gmail.com>]

On Tuesday 06 September 2016 09:42 AM, Jaikiran Pai wrote:
> Hi Todd,
>
>> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
>> SHA1PRNG.
>
> Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our 
> setup. However, right now, (intentionally) we have single broker 
> within the setup and based on the JIRA details, I believe this one 
> won't affect us for now.

I read that JIRA again and saw the stacktrace. I might be wrong when I 
said this probably doesn't affect us given that the code path appears to 
the same for other SSL interactions. I'm going to introduce this patch 
(or in fact the latest 0.10 branch) in our setup and see how it goes.

-Jaikiran


>
>
>>
>> Right now, we're specifically avoiding moving consume traffic to SSL, 
>> due
>> to the zero copy send issue. Now I've been told (but I have not
>> investigated) that OpenSSL can solve this. It would probably be a 
>> good use
>> of time to look into that further.
>
> Unfortunately, the way our application is written, we don't have the 
> ability to enable SSL for producers and disable it for consumers. 
> Changing the application to support something like that is doable, but 
> I will have to make sure that this is one of the reasons for the 
> slowness.
>
>
> Thanks for all your inputs. I was trying to avoid coming up with a 
> reproducer which closely matches our application usage scenario (given 
> our tight delivery timelines for the product), but from what you say 
> and a reply Tom Crayford in this thread, I am starting to feel that 
> not everyone is affected to the extent we are. I will see if I can 
> narrow to this down to something more concrete.
>
> -Jaikiran
>
>
>
>
>>
>> That said, switching the message format to the newer option (KIP-31 I
>> believe?) will result in the brokers not needing to recompress message
>> batches that are produced. This should result in a significant 
>> reduction in
>> CPU usage, which may offset the cost of SSL. We haven't had a chance to
>> fully investigate this, however, as changing that config depends on the
>> clients being updated to support the new format.
>>
>> -Todd
>>
>> On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com> 
>> wrote:
>>
>>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
>>> decided to start using SSL for Kafka communication between broker and
>>> clients. Right now, we have a pretty basic setup with just 1 broker 
>>> with
>>> SSL keystore setup and the Java client(s) communicate using the
>>> Producer/Consumer APIs against this single broker. There's no client 
>>> auth
>>> (intentionally) right now. We also have plain text enabled for the 
>>> initial
>>> testing.
>>>
>>> What we have noticed is that the consumer/producer performance when 
>>> SSL is
>>> enabled is noticeably poor when compared to plain text. I understand 
>>> that
>>> there are expected to be performance impacts when SSL is enabled but 
>>> the
>>> order of magnitude is too high and in fact it shows up in a very 
>>> noticeable
>>> fashion in our product. I do have the numbers, but I haven't been 
>>> able to
>>> narrow it down yet (because there's nothing that stands out, except 
>>> that
>>> it's slow). Our application code is exactly the same between non-SSL 
>>> and
>>> SSL usage.
>>>
>>> Furthermore, I'm aware of this specific JIRA in Kafka
>>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
>>> similar issue. So what I would like to know is, in context of Kafka 
>>> 0.10.x
>>> releases and Java 8 support, are there any timelines that the dev 
>>> team is
>>> looking for in terms of improving this performance issue (which I 
>>> believe
>>> requires usage of OpenSSL or other non-JDK implementations of 
>>> SSLEngine)?
>>> We would like to go for GA of our product in the next couple of 
>>> months and
>>> in order to do that, we do plan to have Kafka over SSL working with
>>> reasonably good performance, but the current performance isn't 
>>> promising.
>>> Expecting this to be fixed in the next couple of months and have it
>>> available in 0.10.x is probably too much to expect, but if we know the
>>> plans around this, we should be able to come up with a plan of our 
>>> own for
>>> our product.
>>>
>>>
>>> -Jaikiran
>>>
>>
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Jaikiran Pai <ja...@gmail.com>]

Hi Todd,


On Sunday 04 September 2016 10:58 PM, Todd Palino wrote:
> We've been using SSL for produce traffic (mirror makers only right now, but
> that's a very large percentage of traffic for us), and we're in the process
> of turning it on for inter broker traffic as well. Our experience is that
> this does not present a significant amount of overhead to the brokers.
> Specifically with switching over the IBP, we were expecting a lot more of a
> hit, and it really only ended up being something like a 5% increase in
> system load, and no reduction in the cluster capacity, in our test cluster.

The slowness we are noticing is on the client JVM which talk to the 
broker. The clients are using (new) Java Kafka producer and consumer 
APIs and the timings that we are seeing in the production and 
consumption of these messages is what is different with and without SSL.


> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
> SHA1PRNG.

Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our 
setup. However, right now, (intentionally) we have single broker within 
the setup and based on the JIRA details, I believe this one won't affect 
us for now.


>
> Right now, we're specifically avoiding moving consume traffic to SSL, due
> to the zero copy send issue. Now I've been told (but I have not
> investigated) that OpenSSL can solve this. It would probably be a good use
> of time to look into that further.

Unfortunately, the way our application is written, we don't have the 
ability to enable SSL for producers and disable it for consumers. 
Changing the application to support something like that is doable, but I 
will have to make sure that this is one of the reasons for the slowness.


Thanks for all your inputs. I was trying to avoid coming up with a 
reproducer which closely matches our application usage scenario (given 
our tight delivery timelines for the product), but from what you say and 
a reply Tom Crayford in this thread, I am starting to feel that not 
everyone is affected to the extent we are. I will see if I can narrow to 
this down to something more concrete.

-Jaikiran




>
> That said, switching the message format to the newer option (KIP-31 I
> believe?) will result in the brokers not needing to recompress message
> batches that are produced. This should result in a significant reduction in
> CPU usage, which may offset the cost of SSL. We haven't had a chance to
> fully investigate this, however, as changing that config depends on the
> clients being updated to support the new format.
>
> -Todd
>
> On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com> wrote:
>
>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
>> decided to start using SSL for Kafka communication between broker and
>> clients. Right now, we have a pretty basic setup with just 1 broker with
>> SSL keystore setup and the Java client(s) communicate using the
>> Producer/Consumer APIs against this single broker. There's no client auth
>> (intentionally) right now. We also have plain text enabled for the initial
>> testing.
>>
>> What we have noticed is that the consumer/producer performance when SSL is
>> enabled is noticeably poor when compared to plain text. I understand that
>> there are expected to be performance impacts when SSL is enabled but the
>> order of magnitude is too high and in fact it shows up in a very noticeable
>> fashion in our product. I do have the numbers, but I haven't been able to
>> narrow it down yet (because there's nothing that stands out, except that
>> it's slow). Our application code is exactly the same between non-SSL and
>> SSL usage.
>>
>> Furthermore, I'm aware of this specific JIRA in Kafka
>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
>> similar issue. So what I would like to know is, in context of Kafka 0.10.x
>> releases and Java 8 support, are there any timelines that the dev team is
>> looking for in terms of improving this performance issue (which I believe
>> requires usage of OpenSSL or other non-JDK implementations of SSLEngine)?
>> We would like to go for GA of our product in the next couple of months and
>> in order to do that, we do plan to have Kafka over SSL working with
>> reasonably good performance, but the current performance isn't promising.
>> Expecting this to be fixed in the next couple of months and have it
>> available in 0.10.x is probably too much to expect, but if we know the
>> plans around this, we should be able to come up with a plan of our own for
>> our product.
>>
>>
>> -Jaikiran
>>
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Jaikiran Pai <ja...@gmail.com>]

Hi Tom,

On Monday 05 September 2016 06:32 PM, Tom Crayford wrote:
> We've been using SSL quite successfully on a few thousand production
> clusters, and have done heavy performance testing, and as far as we can
> see, it doesn't impact the cluster's capacity. The only case I could see
> that happening is if you have a *lot* of connection churn, but that isn't
> very typical for most Kafka usage.

We have a single broker and the clients to this broker are the Java 
producers and consumers across different Java VMs. The producers are one 
per app (as recommended by Kafka) and consumers are around 3 per app 
(all pointing to the same broker, obviously). I don't think this adds to 
any kind of connection churn. Furthermore, the slowness we are seeing is 
in the client JVMs and no real noticeable issues on the broker itself.


>
> That's for both produce and consume traffic. We do see increase amounts of
> garbage collection under high consumer load with SSL enabled, but nothing
> too horrible (brokers are well below the SLA of GC to keep their zookeeper
> sessions alive).

This GC you notice, is it on the broker(s) or on the clients which hosts 
the producer/consumers? Either way, this is a good hint and something I 
hadn't considered in my setup. I'll see if I can get some GC 
logs/metrics on our setup for this. Thanks for the input.


-Jaikiran


>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tp...@gmail.com> wrote:
>
>> We've been using SSL for produce traffic (mirror makers only right now, but
>> that's a very large percentage of traffic for us), and we're in the process
>> of turning it on for inter broker traffic as well. Our experience is that
>> this does not present a significant amount of overhead to the brokers.
>> Specifically with switching over the IBP, we were expecting a lot more of a
>> hit, and it really only ended up being something like a 5% increase in
>> system load, and no reduction in the cluster capacity, in our test cluster.
>> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
>> SHA1PRNG.
>>
>> Right now, we're specifically avoiding moving consume traffic to SSL, due
>> to the zero copy send issue. Now I've been told (but I have not
>> investigated) that OpenSSL can solve this. It would probably be a good use
>> of time to look into that further.
>>
>> That said, switching the message format to the newer option (KIP-31 I
>> believe?) will result in the brokers not needing to recompress message
>> batches that are produced. This should result in a significant reduction in
>> CPU usage, which may offset the cost of SSL. We haven't had a chance to
>> fully investigate this, however, as changing that config depends on the
>> clients being updated to support the new format.
>>
>> -Todd
>>
>> On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
>>> decided to start using SSL for Kafka communication between broker and
>>> clients. Right now, we have a pretty basic setup with just 1 broker with
>>> SSL keystore setup and the Java client(s) communicate using the
>>> Producer/Consumer APIs against this single broker. There's no client auth
>>> (intentionally) right now. We also have plain text enabled for the
>> initial
>>> testing.
>>>
>>> What we have noticed is that the consumer/producer performance when SSL
>> is
>>> enabled is noticeably poor when compared to plain text. I understand that
>>> there are expected to be performance impacts when SSL is enabled but the
>>> order of magnitude is too high and in fact it shows up in a very
>> noticeable
>>> fashion in our product. I do have the numbers, but I haven't been able to
>>> narrow it down yet (because there's nothing that stands out, except that
>>> it's slow). Our application code is exactly the same between non-SSL and
>>> SSL usage.
>>>
>>> Furthermore, I'm aware of this specific JIRA in Kafka
>>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
>>> similar issue. So what I would like to know is, in context of Kafka
>> 0.10.x
>>> releases and Java 8 support, are there any timelines that the dev team is
>>> looking for in terms of improving this performance issue (which I believe
>>> requires usage of OpenSSL or other non-JDK implementations of SSLEngine)?
>>> We would like to go for GA of our product in the next couple of months
>> and
>>> in order to do that, we do plan to have Kafka over SSL working with
>>> reasonably good performance, but the current performance isn't promising.
>>> Expecting this to be fixed in the next couple of months and have it
>>> available in 0.10.x is probably too much to expect, but if we know the
>>> plans around this, we should be able to come up with a plan of our own
>> for
>>> our product.
>>>
>>>
>>> -Jaikiran
>>>
>>
>> --
>> *Todd Palino*
>> Staff Site Reliability Engineer
>> Data Infrastructure Streaming
>>
>>
>>
>> linkedin.com/in/toddpalino
>>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Tom Crayford <tc...@heroku.com>]

We've been using SSL quite successfully on a few thousand production
clusters, and have done heavy performance testing, and as far as we can
see, it doesn't impact the cluster's capacity. The only case I could see
that happening is if you have a *lot* of connection churn, but that isn't
very typical for most Kafka usage.

That's for both produce and consume traffic. We do see increase amounts of
garbage collection under high consumer load with SSL enabled, but nothing
too horrible (brokers are well below the SLA of GC to keep their zookeeper
sessions alive).

Thanks

Tom Crayford
Heroku Kafka

On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tp...@gmail.com> wrote:

> We've been using SSL for produce traffic (mirror makers only right now, but
> that's a very large percentage of traffic for us), and we're in the process
> of turning it on for inter broker traffic as well. Our experience is that
> this does not present a significant amount of overhead to the brokers.
> Specifically with switching over the IBP, we were expecting a lot more of a
> hit, and it really only ended up being something like a 5% increase in
> system load, and no reduction in the cluster capacity, in our test cluster.
> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
> SHA1PRNG.
>
> Right now, we're specifically avoiding moving consume traffic to SSL, due
> to the zero copy send issue. Now I've been told (but I have not
> investigated) that OpenSSL can solve this. It would probably be a good use
> of time to look into that further.
>
> That said, switching the message format to the newer option (KIP-31 I
> believe?) will result in the brokers not needing to recompress message
> batches that are produced. This should result in a significant reduction in
> CPU usage, which may offset the cost of SSL. We haven't had a chance to
> fully investigate this, however, as changing that config depends on the
> clients being updated to support the new format.
>
> -Todd
>
> On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com>
> wrote:
>
> > We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
> > decided to start using SSL for Kafka communication between broker and
> > clients. Right now, we have a pretty basic setup with just 1 broker with
> > SSL keystore setup and the Java client(s) communicate using the
> > Producer/Consumer APIs against this single broker. There's no client auth
> > (intentionally) right now. We also have plain text enabled for the
> initial
> > testing.
> >
> > What we have noticed is that the consumer/producer performance when SSL
> is
> > enabled is noticeably poor when compared to plain text. I understand that
> > there are expected to be performance impacts when SSL is enabled but the
> > order of magnitude is too high and in fact it shows up in a very
> noticeable
> > fashion in our product. I do have the numbers, but I haven't been able to
> > narrow it down yet (because there's nothing that stands out, except that
> > it's slow). Our application code is exactly the same between non-SSL and
> > SSL usage.
> >
> > Furthermore, I'm aware of this specific JIRA in Kafka
> > https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
> > similar issue. So what I would like to know is, in context of Kafka
> 0.10.x
> > releases and Java 8 support, are there any timelines that the dev team is
> > looking for in terms of improving this performance issue (which I believe
> > requires usage of OpenSSL or other non-JDK implementations of SSLEngine)?
> > We would like to go for GA of our product in the next couple of months
> and
> > in order to do that, we do plan to have Kafka over SSL working with
> > reasonably good performance, but the current performance isn't promising.
> > Expecting this to be fixed in the next couple of months and have it
> > available in 0.10.x is probably too much to expect, but if we know the
> > plans around this, we should be able to come up with a plan of our own
> for
> > our product.
> >
> >
> > -Jaikiran
> >
>
>
> --
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Ismael Juma <is...@juma.me.uk>]

Great!

Ismael

On Mon, Sep 4, 2017 at 1:43 PM, Todd Palino <tp...@gmail.com> wrote:

> I will say that we've been turning on TLS consume lately (including using
> it for IBP, which we've been doing for a while) and we haven't seen any of
> the performance concerns that we originally did. Very little hit at all.
>
> -Todd
>
> On Sep 4, 2017 12:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
>
> > By the way, in-kernel TLS has now landed in the Linux kernel:
> >
> > https://github.com/torvalds/linux/blob/master/
> > Documentation/networking/tls.txt
> >
> > There is work in progress to take advantage of that in OpenSSL:
> >
> > https://github.com/Mellanox/tls-openssl
> >
> > Ismael
> >
> >
> > On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > Yeah, that's why I mentioned it with a caveat :) Someone (I can't
> recall
> > > who, but it was someone I consider reasonably knowledgable as I
> actually
> > > gave it some weight) mentioned it, but I haven't looked into it further
> > > than that. I agree that I don't see how this is going to help us at the
> > app
> > > layer.
> > >
> > > -Todd
> > >
> > > On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Todd,
> > > >
> > > > Thanks for sharing your experience enabling TLS in your clusters.
> Very
> > > > helpful. One comment below.
> > > >
> > > > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > > > <javascript:;>> wrote:
> > > > >
> > > > > Right now, we're specifically avoiding moving consume traffic to
> SSL,
> > > due
> > > > > to the zero copy send issue. Now I've been told (but I have not
> > > > > investigated) that OpenSSL can solve this. It would probably be a
> > good
> > > > use
> > > > > of time to look into that further.
> > > > >
> > > >
> > > > As far as I know, OpenSSL can reduce the TLS overhead, but we will
> > still
> > > > lose the zero-copy optimisation. There is some attempts at making it
> > > > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > > > probably too early for us to consider that for Kafka.
> > > >
> > > > Ismael
> > > >
> > > > [1] https://lwn.net/Articles/666509/
> > > > [2]
> > > > http://techblog.netflix.com/2016/08/protecting-netflix-
> > > > viewing-privacy-at.html
> > > >
> > >
> > >
> > > --
> > > *Todd Palino*
> > > Staff Site Reliability Engineer
> > > Data Infrastructure Streaming
> > >
> > >
> > >
> > > linkedin.com/in/toddpalino
> > >
> >
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Todd Palino <tp...@gmail.com>]

I will say that we've been turning on TLS consume lately (including using
it for IBP, which we've been doing for a while) and we haven't seen any of
the performance concerns that we originally did. Very little hit at all.

-Todd

On Sep 4, 2017 12:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:

> By the way, in-kernel TLS has now landed in the Linux kernel:
>
> https://github.com/torvalds/linux/blob/master/
> Documentation/networking/tls.txt
>
> There is work in progress to take advantage of that in OpenSSL:
>
> https://github.com/Mellanox/tls-openssl
>
> Ismael
>
>
> On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:
>
> > Yeah, that's why I mentioned it with a caveat :) Someone (I can't recall
> > who, but it was someone I consider reasonably knowledgable as I actually
> > gave it some weight) mentioned it, but I haven't looked into it further
> > than that. I agree that I don't see how this is going to help us at the
> app
> > layer.
> >
> > -Todd
> >
> > On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Todd,
> > >
> > > Thanks for sharing your experience enabling TLS in your clusters. Very
> > > helpful. One comment below.
> > >
> > > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > > <javascript:;>> wrote:
> > > >
> > > > Right now, we're specifically avoiding moving consume traffic to SSL,
> > due
> > > > to the zero copy send issue. Now I've been told (but I have not
> > > > investigated) that OpenSSL can solve this. It would probably be a
> good
> > > use
> > > > of time to look into that further.
> > > >
> > >
> > > As far as I know, OpenSSL can reduce the TLS overhead, but we will
> still
> > > lose the zero-copy optimisation. There is some attempts at making it
> > > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > > probably too early for us to consider that for Kafka.
> > >
> > > Ismael
> > >
> > > [1] https://lwn.net/Articles/666509/
> > > [2]
> > > http://techblog.netflix.com/2016/08/protecting-netflix-
> > > viewing-privacy-at.html
> > >
> >
> >
> > --
> > *Todd Palino*
> > Staff Site Reliability Engineer
> > Data Infrastructure Streaming
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Ismael Juma <is...@juma.me.uk>]

By the way, in-kernel TLS has now landed in the Linux kernel:

https://github.com/torvalds/linux/blob/master/Documentation/networking/tls.txt

There is work in progress to take advantage of that in OpenSSL:

https://github.com/Mellanox/tls-openssl

Ismael


On Tue, Sep 6, 2016 at 1:48 PM, Todd Palino <tp...@gmail.com> wrote:

> Yeah, that's why I mentioned it with a caveat :) Someone (I can't recall
> who, but it was someone I consider reasonably knowledgable as I actually
> gave it some weight) mentioned it, but I haven't looked into it further
> than that. I agree that I don't see how this is going to help us at the app
> layer.
>
> -Todd
>
> On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Todd,
> >
> > Thanks for sharing your experience enabling TLS in your clusters. Very
> > helpful. One comment below.
> >
> > On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> > <javascript:;>> wrote:
> > >
> > > Right now, we're specifically avoiding moving consume traffic to SSL,
> due
> > > to the zero copy send issue. Now I've been told (but I have not
> > > investigated) that OpenSSL can solve this. It would probably be a good
> > use
> > > of time to look into that further.
> > >
> >
> > As far as I know, OpenSSL can reduce the TLS overhead, but we will still
> > lose the zero-copy optimisation. There is some attempts at making it
> > possible to retain zero-copy with TLS in the kernel[1][2], but it's
> > probably too early for us to consider that for Kafka.
> >
> > Ismael
> >
> > [1] https://lwn.net/Articles/666509/
> > [2]
> > http://techblog.netflix.com/2016/08/protecting-netflix-
> > viewing-privacy-at.html
> >
>
>
> --
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Todd Palino <tp...@gmail.com>]

Yeah, that's why I mentioned it with a caveat :) Someone (I can't recall
who, but it was someone I consider reasonably knowledgable as I actually
gave it some weight) mentioned it, but I haven't looked into it further
than that. I agree that I don't see how this is going to help us at the app
layer.

-Todd

On Tuesday, September 6, 2016, Ismael Juma <is...@juma.me.uk> wrote:

> Hi Todd,
>
> Thanks for sharing your experience enabling TLS in your clusters. Very
> helpful. One comment below.
>
> On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tpalino@gmail.com
> <javascript:;>> wrote:
> >
> > Right now, we're specifically avoiding moving consume traffic to SSL, due
> > to the zero copy send issue. Now I've been told (but I have not
> > investigated) that OpenSSL can solve this. It would probably be a good
> use
> > of time to look into that further.
> >
>
> As far as I know, OpenSSL can reduce the TLS overhead, but we will still
> lose the zero-copy optimisation. There is some attempts at making it
> possible to retain zero-copy with TLS in the kernel[1][2], but it's
> probably too early for us to consider that for Kafka.
>
> Ismael
>
> [1] https://lwn.net/Articles/666509/
> [2]
> http://techblog.netflix.com/2016/08/protecting-netflix-
> viewing-privacy-at.html
>


-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Ismael Juma <is...@juma.me.uk>]

Hi Todd,

Thanks for sharing your experience enabling TLS in your clusters. Very
helpful. One comment below.

On Sun, Sep 4, 2016 at 6:28 PM, Todd Palino <tp...@gmail.com> wrote:
>
> Right now, we're specifically avoiding moving consume traffic to SSL, due
> to the zero copy send issue. Now I've been told (but I have not
> investigated) that OpenSSL can solve this. It would probably be a good use
> of time to look into that further.
>

As far as I know, OpenSSL can reduce the TLS overhead, but we will still
lose the zero-copy optimisation. There is some attempts at making it
possible to retain zero-copy with TLS in the kernel[1][2], but it's
probably too early for us to consider that for Kafka.

Ismael

[1] https://lwn.net/Articles/666509/
[2]
http://techblog.netflix.com/2016/08/protecting-netflix-viewing-privacy-at.html

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

[Todd Palino <tp...@gmail.com>]

We've been using SSL for produce traffic (mirror makers only right now, but
that's a very large percentage of traffic for us), and we're in the process
of turning it on for inter broker traffic as well. Our experience is that
this does not present a significant amount of overhead to the brokers.
Specifically with switching over the IBP, we were expecting a lot more of a
hit, and it really only ended up being something like a 5% increase in
system load, and no reduction in the cluster capacity, in our test cluster.
Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
SHA1PRNG.

Right now, we're specifically avoiding moving consume traffic to SSL, due
to the zero copy send issue. Now I've been told (but I have not
investigated) that OpenSSL can solve this. It would probably be a good use
of time to look into that further.

That said, switching the message format to the newer option (KIP-31 I
believe?) will result in the brokers not needing to recompress message
batches that are produced. This should result in a significant reduction in
CPU usage, which may offset the cost of SSL. We haven't had a chance to
fully investigate this, however, as changing that config depends on the
clients being updated to support the new format.

-Todd

On Sunday, September 4, 2016, Jaikiran Pai <ja...@gmail.com> wrote:

> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
> decided to start using SSL for Kafka communication between broker and
> clients. Right now, we have a pretty basic setup with just 1 broker with
> SSL keystore setup and the Java client(s) communicate using the
> Producer/Consumer APIs against this single broker. There's no client auth
> (intentionally) right now. We also have plain text enabled for the initial
> testing.
>
> What we have noticed is that the consumer/producer performance when SSL is
> enabled is noticeably poor when compared to plain text. I understand that
> there are expected to be performance impacts when SSL is enabled but the
> order of magnitude is too high and in fact it shows up in a very noticeable
> fashion in our product. I do have the numbers, but I haven't been able to
> narrow it down yet (because there's nothing that stands out, except that
> it's slow). Our application code is exactly the same between non-SSL and
> SSL usage.
>
> Furthermore, I'm aware of this specific JIRA in Kafka
> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
> similar issue. So what I would like to know is, in context of Kafka 0.10.x
> releases and Java 8 support, are there any timelines that the dev team is
> looking for in terms of improving this performance issue (which I believe
> requires usage of OpenSSL or other non-JDK implementations of SSLEngine)?
> We would like to go for GA of our product in the next couple of months and
> in order to do that, we do plan to have Kafka over SSL working with
> reasonably good performance, but the current performance isn't promising.
> Expecting this to be fixed in the next couple of months and have it
> available in 0.10.x is probably too much to expect, but if we know the
> plans around this, we should be able to come up with a plan of our own for
> our product.
>
>
> -Jaikiran
>


-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino