You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jaikiran Pai <ja...@gmail.com> on 2017/10/30 08:08:36 UTC
Experimenting with Kafka and OpenSSL
We have been using Kafka in some of our projects for the past couple of
years. Our experience with Kafka and SSL had shown some performance
issues when we had seriously tested it (which admittedly was around a
year back). Our basic tests did show that things had improved over time
with newer versions, but we didn't get a chance to fully test and move
to SSL for Kafka.
Incidentally, I happened to be looking into some other things related to
SSL and decided to experiment with using openssl as the SSL provider for
Kafka. I had heard OpenSSL performs better than the engine shipped
default in JRE, but hadn't ever got a chance to do any experiments. This
past few weeks, I decided to spend some time trying it. I have noted the
experimentation and the performance numbers in my blog[1]. The initial
basic performance testing (using the scripts shipped in Kafka) does show
promising improvements. Like I note in my blog, this was a very basic
performance test just to see if OpenSSL can be pursued as an option
(both in terms of being functional and performant) if we do decide to.
I know some of the members in these lists do extensive performance
testing with Kafka (and SSL), so I thought I will bring this to their
notice.
[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
-Jaikiran
Re: Experimenting with Kafka and OpenSSL
Posted by Ismael Juma <is...@gmail.com>.
Thanks Jaikiran. This is is useful. However, as you point out, the ciphers
are an important factor. It would be good to ensure that the encryption
strength is comparable to make it fair. The first step could be to output
the ciphers used by Java 9 and OpenSSL by default.
Ismael
On 11 Nov 2017 7:37 am, "Jaikiran Pai" <ja...@gmail.com> wrote:
I ran these same tests with Java 9 runtime today and have updated the blog
to include these numbers[1]. I'm pasting the summary for Java 9 here:
- Both for producer and consumer, there's a *drastic improvement in the JRE
shipped SSLEngine numbers, in almost all metrics, in Java 9 as compared to
its counterpart in Java 8*. It's especially prominent in messages with
higher sizes.
- There's not much difference in the numbers for WildFly OpenSSL, in Java
9, as compared to its Java 8 counterpart. In fact, the consumer performance
numbers of WildFly OpenSSL in Java 9 have dropped slightly when compared to
Java 8. The producer performance in Java 9 with WildFly OpenSSL have
however improved slightly when compared to Java 8.
- When the numbers of producer and consumer metrics of WildFly OpenSSL with
Java 9 runtime are compared with the JRE shipped SSL engine in Java 9,
*WildFly OpenSSL still out-performs the one shipped in JRE*.
Like in the Java 8 runs, all default configs and settings were used, not
just for Kafka but even the JRE (i.e. no explicit choice of cipher suites).
[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
-Jaikiran
On 30/10/17 5:33 PM, Ismael Juma wrote:
> If Java 9 is used by both clients and brokers, AES GCM is used by default.
> I did a quick test a while back and there was a significant improvement:
>
> https://twitter.com/ijuma/status/905847523897724929
>
> Ismael
>
> On Mon, Oct 30, 2017 at 11:57 AM, Radu Radutiu <rr...@gmail.com> wrote:
>
> If you test with Java 9 please make sure to use an accelerated cipher suite
>> (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
>>
>> Radu
>>
>> On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>> I haven't yet had a chance to try out Java 9, but that's definitely on my
>>> TODO list, maybe sometime this weekend.
>>>
>>> Thanks for pointing me to KAFKA-2561. I had missed that.
>>>
>>> -Jaikiran
>>>
>>>
>>>
>>> On 30/10/17 4:17 PM, Mickael Maison wrote:
>>>
>>> Thanks for sharing, very interesting read.
>>>>
>>>> Did you get a chance to try JDK 9 ?
>>>>
>>>> We also considered using OpenSSL instead of JSSE especially since
>>>> Netty made an easy to re-use package (netty-tcnative).
>>>>
>>>> There was KAFKA-2561
>>>> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
>>>> a few numbers and what would be need to get it working.
>>>>
>>>> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <jai.forums2013@gmail.com
>>>> wrote:
>>>>
>>>> We have been using Kafka in some of our projects for the past couple of
>>>>> years. Our experience with Kafka and SSL had shown some performance
>>>>> issues
>>>>> when we had seriously tested it (which admittedly was around a year
>>>>> back).
>>>>> Our basic tests did show that things had improved over time with newer
>>>>> versions, but we didn't get a chance to fully test and move to SSL for
>>>>> Kafka.
>>>>>
>>>>> Incidentally, I happened to be looking into some other things related
>>>>>
>>>> to
>>
>>> SSL
>>>>> and decided to experiment with using openssl as the SSL provider for
>>>>> Kafka.
>>>>> I had heard OpenSSL performs better than the engine shipped default in
>>>>> JRE,
>>>>> but hadn't ever got a chance to do any experiments. This past few
>>>>>
>>>> weeks,
>>
>>> I
>>>>> decided to spend some time trying it. I have noted the experimentation
>>>>> and
>>>>> the performance numbers in my blog[1]. The initial basic performance
>>>>> testing
>>>>> (using the scripts shipped in Kafka) does show promising improvements.
>>>>> Like
>>>>> I note in my blog, this was a very basic performance test just to see
>>>>>
>>>> if
>>
>>> OpenSSL can be pursued as an option (both in terms of being functional
>>>>> and
>>>>> performant) if we do decide to.
>>>>>
>>>>> I know some of the members in these lists do extensive performance
>>>>> testing
>>>>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>>>>
>>>>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-
>>>>>
>>>> with-openssl.html
>>
>>> -Jaikiran
>>>>>
>>>>>
>>>>>
Re: Experimenting with Kafka and OpenSSL
Posted by Ismael Juma <is...@gmail.com>.
Thanks Jaikiran. This is is useful. However, as you point out, the ciphers
are an important factor. It would be good to ensure that the encryption
strength is comparable to make it fair. The first step could be to output
the ciphers used by Java 9 and OpenSSL by default.
Ismael
On 11 Nov 2017 7:37 am, "Jaikiran Pai" <ja...@gmail.com> wrote:
I ran these same tests with Java 9 runtime today and have updated the blog
to include these numbers[1]. I'm pasting the summary for Java 9 here:
- Both for producer and consumer, there's a *drastic improvement in the JRE
shipped SSLEngine numbers, in almost all metrics, in Java 9 as compared to
its counterpart in Java 8*. It's especially prominent in messages with
higher sizes.
- There's not much difference in the numbers for WildFly OpenSSL, in Java
9, as compared to its Java 8 counterpart. In fact, the consumer performance
numbers of WildFly OpenSSL in Java 9 have dropped slightly when compared to
Java 8. The producer performance in Java 9 with WildFly OpenSSL have
however improved slightly when compared to Java 8.
- When the numbers of producer and consumer metrics of WildFly OpenSSL with
Java 9 runtime are compared with the JRE shipped SSL engine in Java 9,
*WildFly OpenSSL still out-performs the one shipped in JRE*.
Like in the Java 8 runs, all default configs and settings were used, not
just for Kafka but even the JRE (i.e. no explicit choice of cipher suites).
[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
-Jaikiran
On 30/10/17 5:33 PM, Ismael Juma wrote:
> If Java 9 is used by both clients and brokers, AES GCM is used by default.
> I did a quick test a while back and there was a significant improvement:
>
> https://twitter.com/ijuma/status/905847523897724929
>
> Ismael
>
> On Mon, Oct 30, 2017 at 11:57 AM, Radu Radutiu <rr...@gmail.com> wrote:
>
> If you test with Java 9 please make sure to use an accelerated cipher suite
>> (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
>>
>> Radu
>>
>> On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>> I haven't yet had a chance to try out Java 9, but that's definitely on my
>>> TODO list, maybe sometime this weekend.
>>>
>>> Thanks for pointing me to KAFKA-2561. I had missed that.
>>>
>>> -Jaikiran
>>>
>>>
>>>
>>> On 30/10/17 4:17 PM, Mickael Maison wrote:
>>>
>>> Thanks for sharing, very interesting read.
>>>>
>>>> Did you get a chance to try JDK 9 ?
>>>>
>>>> We also considered using OpenSSL instead of JSSE especially since
>>>> Netty made an easy to re-use package (netty-tcnative).
>>>>
>>>> There was KAFKA-2561
>>>> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
>>>> a few numbers and what would be need to get it working.
>>>>
>>>> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <jai.forums2013@gmail.com
>>>> wrote:
>>>>
>>>> We have been using Kafka in some of our projects for the past couple of
>>>>> years. Our experience with Kafka and SSL had shown some performance
>>>>> issues
>>>>> when we had seriously tested it (which admittedly was around a year
>>>>> back).
>>>>> Our basic tests did show that things had improved over time with newer
>>>>> versions, but we didn't get a chance to fully test and move to SSL for
>>>>> Kafka.
>>>>>
>>>>> Incidentally, I happened to be looking into some other things related
>>>>>
>>>> to
>>
>>> SSL
>>>>> and decided to experiment with using openssl as the SSL provider for
>>>>> Kafka.
>>>>> I had heard OpenSSL performs better than the engine shipped default in
>>>>> JRE,
>>>>> but hadn't ever got a chance to do any experiments. This past few
>>>>>
>>>> weeks,
>>
>>> I
>>>>> decided to spend some time trying it. I have noted the experimentation
>>>>> and
>>>>> the performance numbers in my blog[1]. The initial basic performance
>>>>> testing
>>>>> (using the scripts shipped in Kafka) does show promising improvements.
>>>>> Like
>>>>> I note in my blog, this was a very basic performance test just to see
>>>>>
>>>> if
>>
>>> OpenSSL can be pursued as an option (both in terms of being functional
>>>>> and
>>>>> performant) if we do decide to.
>>>>>
>>>>> I know some of the members in these lists do extensive performance
>>>>> testing
>>>>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>>>>
>>>>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-
>>>>>
>>>> with-openssl.html
>>
>>> -Jaikiran
>>>>>
>>>>>
>>>>>
Re: Experimenting with Kafka and OpenSSL
Posted by Jaikiran Pai <ja...@gmail.com>.
I ran these same tests with Java 9 runtime today and have updated the
blog to include these numbers[1]. I'm pasting the summary for Java 9 here:
- Both for producer and consumer, there's a *drastic improvement in the
JRE shipped SSLEngine numbers, in almost all metrics, in Java 9 as
compared to its counterpart in Java 8*. It's especially prominent in
messages with higher sizes.
- There's not much difference in the numbers for WildFly OpenSSL, in
Java 9, as compared to its Java 8 counterpart. In fact, the consumer
performance numbers of WildFly OpenSSL in Java 9 have dropped slightly
when compared to Java 8. The producer performance in Java 9 with WildFly
OpenSSL have however improved slightly when compared to Java 8.
- When the numbers of producer and consumer metrics of WildFly OpenSSL
with Java 9 runtime are compared with the JRE shipped SSL engine in Java
9, *WildFly OpenSSL still out-performs the one shipped in JRE*.
Like in the Java 8 runs, all default configs and settings were used, not
just for Kafka but even the JRE (i.e. no explicit choice of cipher suites).
[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
-Jaikiran
On 30/10/17 5:33 PM, Ismael Juma wrote:
> If Java 9 is used by both clients and brokers, AES GCM is used by default.
> I did a quick test a while back and there was a significant improvement:
>
> https://twitter.com/ijuma/status/905847523897724929
>
> Ismael
>
> On Mon, Oct 30, 2017 at 11:57 AM, Radu Radutiu <rr...@gmail.com> wrote:
>
>> If you test with Java 9 please make sure to use an accelerated cipher suite
>> (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
>>
>> Radu
>>
>> On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>>> I haven't yet had a chance to try out Java 9, but that's definitely on my
>>> TODO list, maybe sometime this weekend.
>>>
>>> Thanks for pointing me to KAFKA-2561. I had missed that.
>>>
>>> -Jaikiran
>>>
>>>
>>>
>>> On 30/10/17 4:17 PM, Mickael Maison wrote:
>>>
>>>> Thanks for sharing, very interesting read.
>>>>
>>>> Did you get a chance to try JDK 9 ?
>>>>
>>>> We also considered using OpenSSL instead of JSSE especially since
>>>> Netty made an easy to re-use package (netty-tcnative).
>>>>
>>>> There was KAFKA-2561
>>>> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
>>>> a few numbers and what would be need to get it working.
>>>>
>>>> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <jai.forums2013@gmail.com
>>>> wrote:
>>>>
>>>>> We have been using Kafka in some of our projects for the past couple of
>>>>> years. Our experience with Kafka and SSL had shown some performance
>>>>> issues
>>>>> when we had seriously tested it (which admittedly was around a year
>>>>> back).
>>>>> Our basic tests did show that things had improved over time with newer
>>>>> versions, but we didn't get a chance to fully test and move to SSL for
>>>>> Kafka.
>>>>>
>>>>> Incidentally, I happened to be looking into some other things related
>> to
>>>>> SSL
>>>>> and decided to experiment with using openssl as the SSL provider for
>>>>> Kafka.
>>>>> I had heard OpenSSL performs better than the engine shipped default in
>>>>> JRE,
>>>>> but hadn't ever got a chance to do any experiments. This past few
>> weeks,
>>>>> I
>>>>> decided to spend some time trying it. I have noted the experimentation
>>>>> and
>>>>> the performance numbers in my blog[1]. The initial basic performance
>>>>> testing
>>>>> (using the scripts shipped in Kafka) does show promising improvements.
>>>>> Like
>>>>> I note in my blog, this was a very basic performance test just to see
>> if
>>>>> OpenSSL can be pursued as an option (both in terms of being functional
>>>>> and
>>>>> performant) if we do decide to.
>>>>>
>>>>> I know some of the members in these lists do extensive performance
>>>>> testing
>>>>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>>>>
>>>>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-
>> with-openssl.html
>>>>> -Jaikiran
>>>>>
>>>>>
Re: Experimenting with Kafka and OpenSSL
Posted by Jaikiran Pai <ja...@gmail.com>.
I ran these same tests with Java 9 runtime today and have updated the
blog to include these numbers[1]. I'm pasting the summary for Java 9 here:
- Both for producer and consumer, there's a *drastic improvement in the
JRE shipped SSLEngine numbers, in almost all metrics, in Java 9 as
compared to its counterpart in Java 8*. It's especially prominent in
messages with higher sizes.
- There's not much difference in the numbers for WildFly OpenSSL, in
Java 9, as compared to its Java 8 counterpart. In fact, the consumer
performance numbers of WildFly OpenSSL in Java 9 have dropped slightly
when compared to Java 8. The producer performance in Java 9 with WildFly
OpenSSL have however improved slightly when compared to Java 8.
- When the numbers of producer and consumer metrics of WildFly OpenSSL
with Java 9 runtime are compared with the JRE shipped SSL engine in Java
9, *WildFly OpenSSL still out-performs the one shipped in JRE*.
Like in the Java 8 runs, all default configs and settings were used, not
just for Kafka but even the JRE (i.e. no explicit choice of cipher suites).
[1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
-Jaikiran
On 30/10/17 5:33 PM, Ismael Juma wrote:
> If Java 9 is used by both clients and brokers, AES GCM is used by default.
> I did a quick test a while back and there was a significant improvement:
>
> https://twitter.com/ijuma/status/905847523897724929
>
> Ismael
>
> On Mon, Oct 30, 2017 at 11:57 AM, Radu Radutiu <rr...@gmail.com> wrote:
>
>> If you test with Java 9 please make sure to use an accelerated cipher suite
>> (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
>>
>> Radu
>>
>> On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>>> I haven't yet had a chance to try out Java 9, but that's definitely on my
>>> TODO list, maybe sometime this weekend.
>>>
>>> Thanks for pointing me to KAFKA-2561. I had missed that.
>>>
>>> -Jaikiran
>>>
>>>
>>>
>>> On 30/10/17 4:17 PM, Mickael Maison wrote:
>>>
>>>> Thanks for sharing, very interesting read.
>>>>
>>>> Did you get a chance to try JDK 9 ?
>>>>
>>>> We also considered using OpenSSL instead of JSSE especially since
>>>> Netty made an easy to re-use package (netty-tcnative).
>>>>
>>>> There was KAFKA-2561
>>>> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
>>>> a few numbers and what would be need to get it working.
>>>>
>>>> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <jai.forums2013@gmail.com
>>>> wrote:
>>>>
>>>>> We have been using Kafka in some of our projects for the past couple of
>>>>> years. Our experience with Kafka and SSL had shown some performance
>>>>> issues
>>>>> when we had seriously tested it (which admittedly was around a year
>>>>> back).
>>>>> Our basic tests did show that things had improved over time with newer
>>>>> versions, but we didn't get a chance to fully test and move to SSL for
>>>>> Kafka.
>>>>>
>>>>> Incidentally, I happened to be looking into some other things related
>> to
>>>>> SSL
>>>>> and decided to experiment with using openssl as the SSL provider for
>>>>> Kafka.
>>>>> I had heard OpenSSL performs better than the engine shipped default in
>>>>> JRE,
>>>>> but hadn't ever got a chance to do any experiments. This past few
>> weeks,
>>>>> I
>>>>> decided to spend some time trying it. I have noted the experimentation
>>>>> and
>>>>> the performance numbers in my blog[1]. The initial basic performance
>>>>> testing
>>>>> (using the scripts shipped in Kafka) does show promising improvements.
>>>>> Like
>>>>> I note in my blog, this was a very basic performance test just to see
>> if
>>>>> OpenSSL can be pursued as an option (both in terms of being functional
>>>>> and
>>>>> performant) if we do decide to.
>>>>>
>>>>> I know some of the members in these lists do extensive performance
>>>>> testing
>>>>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>>>>
>>>>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-
>> with-openssl.html
>>>>> -Jaikiran
>>>>>
>>>>>
Re: Experimenting with Kafka and OpenSSL
Posted by Ismael Juma <is...@juma.me.uk>.
If Java 9 is used by both clients and brokers, AES GCM is used by default.
I did a quick test a while back and there was a significant improvement:
https://twitter.com/ijuma/status/905847523897724929
Ismael
On Mon, Oct 30, 2017 at 11:57 AM, Radu Radutiu <rr...@gmail.com> wrote:
> If you test with Java 9 please make sure to use an accelerated cipher suite
> (e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
>
> Radu
>
> On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
> wrote:
>
> > I haven't yet had a chance to try out Java 9, but that's definitely on my
> > TODO list, maybe sometime this weekend.
> >
> > Thanks for pointing me to KAFKA-2561. I had missed that.
> >
> > -Jaikiran
> >
> >
> >
> > On 30/10/17 4:17 PM, Mickael Maison wrote:
> >
> >> Thanks for sharing, very interesting read.
> >>
> >> Did you get a chance to try JDK 9 ?
> >>
> >> We also considered using OpenSSL instead of JSSE especially since
> >> Netty made an easy to re-use package (netty-tcnative).
> >>
> >> There was KAFKA-2561
> >> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
> >> a few numbers and what would be need to get it working.
> >>
> >> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <jai.forums2013@gmail.com
> >
> >> wrote:
> >>
> >>> We have been using Kafka in some of our projects for the past couple of
> >>> years. Our experience with Kafka and SSL had shown some performance
> >>> issues
> >>> when we had seriously tested it (which admittedly was around a year
> >>> back).
> >>> Our basic tests did show that things had improved over time with newer
> >>> versions, but we didn't get a chance to fully test and move to SSL for
> >>> Kafka.
> >>>
> >>> Incidentally, I happened to be looking into some other things related
> to
> >>> SSL
> >>> and decided to experiment with using openssl as the SSL provider for
> >>> Kafka.
> >>> I had heard OpenSSL performs better than the engine shipped default in
> >>> JRE,
> >>> but hadn't ever got a chance to do any experiments. This past few
> weeks,
> >>> I
> >>> decided to spend some time trying it. I have noted the experimentation
> >>> and
> >>> the performance numbers in my blog[1]. The initial basic performance
> >>> testing
> >>> (using the scripts shipped in Kafka) does show promising improvements.
> >>> Like
> >>> I note in my blog, this was a very basic performance test just to see
> if
> >>> OpenSSL can be pursued as an option (both in terms of being functional
> >>> and
> >>> performant) if we do decide to.
> >>>
> >>> I know some of the members in these lists do extensive performance
> >>> testing
> >>> with Kafka (and SSL), so I thought I will bring this to their notice.
> >>>
> >>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-
> with-openssl.html
> >>>
> >>> -Jaikiran
> >>>
> >>>
> >
>
Re: Experimenting with Kafka and OpenSSL
Posted by Radu Radutiu <rr...@gmail.com>.
If you test with Java 9 please make sure to use an accelerated cipher suite
(e.g. one that uses AES GCM such as TLS_RSA_WITH_AES_128_GCM_SHA256).
Radu
On Mon, Oct 30, 2017 at 1:49 PM, Jaikiran Pai <ja...@gmail.com>
wrote:
> I haven't yet had a chance to try out Java 9, but that's definitely on my
> TODO list, maybe sometime this weekend.
>
> Thanks for pointing me to KAFKA-2561. I had missed that.
>
> -Jaikiran
>
>
>
> On 30/10/17 4:17 PM, Mickael Maison wrote:
>
>> Thanks for sharing, very interesting read.
>>
>> Did you get a chance to try JDK 9 ?
>>
>> We also considered using OpenSSL instead of JSSE especially since
>> Netty made an easy to re-use package (netty-tcnative).
>>
>> There was KAFKA-2561
>> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
>> a few numbers and what would be need to get it working.
>>
>> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>>> We have been using Kafka in some of our projects for the past couple of
>>> years. Our experience with Kafka and SSL had shown some performance
>>> issues
>>> when we had seriously tested it (which admittedly was around a year
>>> back).
>>> Our basic tests did show that things had improved over time with newer
>>> versions, but we didn't get a chance to fully test and move to SSL for
>>> Kafka.
>>>
>>> Incidentally, I happened to be looking into some other things related to
>>> SSL
>>> and decided to experiment with using openssl as the SSL provider for
>>> Kafka.
>>> I had heard OpenSSL performs better than the engine shipped default in
>>> JRE,
>>> but hadn't ever got a chance to do any experiments. This past few weeks,
>>> I
>>> decided to spend some time trying it. I have noted the experimentation
>>> and
>>> the performance numbers in my blog[1]. The initial basic performance
>>> testing
>>> (using the scripts shipped in Kafka) does show promising improvements.
>>> Like
>>> I note in my blog, this was a very basic performance test just to see if
>>> OpenSSL can be pursued as an option (both in terms of being functional
>>> and
>>> performant) if we do decide to.
>>>
>>> I know some of the members in these lists do extensive performance
>>> testing
>>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>>
>>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>>>
>>> -Jaikiran
>>>
>>>
>
Re: Experimenting with Kafka and OpenSSL
Posted by Jaikiran Pai <ja...@gmail.com>.
I haven't yet had a chance to try out Java 9, but that's definitely on
my TODO list, maybe sometime this weekend.
Thanks for pointing me to KAFKA-2561. I had missed that.
-Jaikiran
On 30/10/17 4:17 PM, Mickael Maison wrote:
> Thanks for sharing, very interesting read.
>
> Did you get a chance to try JDK 9 ?
>
> We also considered using OpenSSL instead of JSSE especially since
> Netty made an easy to re-use package (netty-tcnative).
>
> There was KAFKA-2561
> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
> a few numbers and what would be need to get it working.
>
> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <ja...@gmail.com> wrote:
>> We have been using Kafka in some of our projects for the past couple of
>> years. Our experience with Kafka and SSL had shown some performance issues
>> when we had seriously tested it (which admittedly was around a year back).
>> Our basic tests did show that things had improved over time with newer
>> versions, but we didn't get a chance to fully test and move to SSL for
>> Kafka.
>>
>> Incidentally, I happened to be looking into some other things related to SSL
>> and decided to experiment with using openssl as the SSL provider for Kafka.
>> I had heard OpenSSL performs better than the engine shipped default in JRE,
>> but hadn't ever got a chance to do any experiments. This past few weeks, I
>> decided to spend some time trying it. I have noted the experimentation and
>> the performance numbers in my blog[1]. The initial basic performance testing
>> (using the scripts shipped in Kafka) does show promising improvements. Like
>> I note in my blog, this was a very basic performance test just to see if
>> OpenSSL can be pursued as an option (both in terms of being functional and
>> performant) if we do decide to.
>>
>> I know some of the members in these lists do extensive performance testing
>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>
>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>>
>> -Jaikiran
>>
Re: Experimenting with Kafka and OpenSSL
Posted by Jaikiran Pai <ja...@gmail.com>.
I haven't yet had a chance to try out Java 9, but that's definitely on
my TODO list, maybe sometime this weekend.
Thanks for pointing me to KAFKA-2561. I had missed that.
-Jaikiran
On 30/10/17 4:17 PM, Mickael Maison wrote:
> Thanks for sharing, very interesting read.
>
> Did you get a chance to try JDK 9 ?
>
> We also considered using OpenSSL instead of JSSE especially since
> Netty made an easy to re-use package (netty-tcnative).
>
> There was KAFKA-2561
> (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
> a few numbers and what would be need to get it working.
>
> On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <ja...@gmail.com> wrote:
>> We have been using Kafka in some of our projects for the past couple of
>> years. Our experience with Kafka and SSL had shown some performance issues
>> when we had seriously tested it (which admittedly was around a year back).
>> Our basic tests did show that things had improved over time with newer
>> versions, but we didn't get a chance to fully test and move to SSL for
>> Kafka.
>>
>> Incidentally, I happened to be looking into some other things related to SSL
>> and decided to experiment with using openssl as the SSL provider for Kafka.
>> I had heard OpenSSL performs better than the engine shipped default in JRE,
>> but hadn't ever got a chance to do any experiments. This past few weeks, I
>> decided to spend some time trying it. I have noted the experimentation and
>> the performance numbers in my blog[1]. The initial basic performance testing
>> (using the scripts shipped in Kafka) does show promising improvements. Like
>> I note in my blog, this was a very basic performance test just to see if
>> OpenSSL can be pursued as an option (both in terms of being functional and
>> performant) if we do decide to.
>>
>> I know some of the members in these lists do extensive performance testing
>> with Kafka (and SSL), so I thought I will bring this to their notice.
>>
>> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>>
>> -Jaikiran
>>
Re: Experimenting with Kafka and OpenSSL
Posted by Mickael Maison <mi...@gmail.com>.
Thanks for sharing, very interesting read.
Did you get a chance to try JDK 9 ?
We also considered using OpenSSL instead of JSSE especially since
Netty made an easy to re-use package (netty-tcnative).
There was KAFKA-2561
(https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
a few numbers and what would be need to get it working.
On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <ja...@gmail.com> wrote:
> We have been using Kafka in some of our projects for the past couple of
> years. Our experience with Kafka and SSL had shown some performance issues
> when we had seriously tested it (which admittedly was around a year back).
> Our basic tests did show that things had improved over time with newer
> versions, but we didn't get a chance to fully test and move to SSL for
> Kafka.
>
> Incidentally, I happened to be looking into some other things related to SSL
> and decided to experiment with using openssl as the SSL provider for Kafka.
> I had heard OpenSSL performs better than the engine shipped default in JRE,
> but hadn't ever got a chance to do any experiments. This past few weeks, I
> decided to spend some time trying it. I have noted the experimentation and
> the performance numbers in my blog[1]. The initial basic performance testing
> (using the scripts shipped in Kafka) does show promising improvements. Like
> I note in my blog, this was a very basic performance test just to see if
> OpenSSL can be pursued as an option (both in terms of being functional and
> performant) if we do decide to.
>
> I know some of the members in these lists do extensive performance testing
> with Kafka (and SSL), so I thought I will bring this to their notice.
>
> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>
> -Jaikiran
>
Re: Experimenting with Kafka and OpenSSL
Posted by Mickael Maison <mi...@gmail.com>.
Thanks for sharing, very interesting read.
Did you get a chance to try JDK 9 ?
We also considered using OpenSSL instead of JSSE especially since
Netty made an easy to re-use package (netty-tcnative).
There was KAFKA-2561
(https://issues.apache.org/jira/browse/KAFKA-2561) where people shared
a few numbers and what would be need to get it working.
On Mon, Oct 30, 2017 at 8:08 AM, Jaikiran Pai <ja...@gmail.com> wrote:
> We have been using Kafka in some of our projects for the past couple of
> years. Our experience with Kafka and SSL had shown some performance issues
> when we had seriously tested it (which admittedly was around a year back).
> Our basic tests did show that things had improved over time with newer
> versions, but we didn't get a chance to fully test and move to SSL for
> Kafka.
>
> Incidentally, I happened to be looking into some other things related to SSL
> and decided to experiment with using openssl as the SSL provider for Kafka.
> I had heard OpenSSL performs better than the engine shipped default in JRE,
> but hadn't ever got a chance to do any experiments. This past few weeks, I
> decided to spend some time trying it. I have noted the experimentation and
> the performance numbers in my blog[1]. The initial basic performance testing
> (using the scripts shipped in Kafka) does show promising improvements. Like
> I note in my blog, this was a very basic performance test just to see if
> OpenSSL can be pursued as an option (both in terms of being functional and
> performant) if we do decide to.
>
> I know some of the members in these lists do extensive performance testing
> with Kafka (and SSL), so I thought I will bring this to their notice.
>
> [1] https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html
>
> -Jaikiran
>