[jira] [Comment Edited] (KAFKA-2561) Optionally support OpenSSL for SSL/TLS

["Ismael Juma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902896#comment-14902896 ] 

Ismael Juma edited comment on KAFKA-2561 at 9/24/15 2:37 PM:
-------------------------------------------------------------

In order to implement this properly (as opposed to a simple test), the following steps are needed:

1. Add an optional build dependency on netty-tcnative. This library contains a fork of tomcat native that is available in Maven and is available for major platforms (Linux, OS X, Windows). It also handles extracting the platform-specific JNI code at runtime (similar to snappy-java). apr and openssl need to be installed separately.
2. Provide an implementation of `SSLEngine` based on OpenSSL. The easy option would be to add an optional dependency on `netty-handler`, which includes this. If this is not acceptable, there are some alternatives like extracting the code into a separate library or copying it into Kafka.
3. Add a way to configure the `SSLEngine` implementation (OpenSSL or JDK).
4. Change `SSLFactory` to build the appropriate `SSLEngine` based on the configuration added in `3`.
5. Potentially introduce a runtime mechanism to select `OpenSslEngine` by default if the required libraries are present (since it's much faster)
6. Potentially update `SSLTransportLayer` to handle differences in behaviour between the different `SSLEngine` implementations (the need for this depends on whether we the issues reported to Netty are fixed or not). In addition to the two issues mentioned in the description, there is also an issue related to the size of the `applicationReadBuffer`.
7. Use `allocateDirect` to allocate the buffers in `SSLTransportLayer` when using `OpenSslEngine` to avoid copies on each `wrap` and `unwrap` call.
8. Design and implement the story around the formats for keys, certificates, key chains and certificate chains supported. OpenSSL doesn't understand the JKS format since it's Java-specific. Netty uses the `PKCS#8` format for keys and PEM format for chains when the OpenSSL engine is used.
9. Update tests to test all `SSLEngine` implementations.

Testing of this is more complicated than usual due to the native code aspect and we would have to test it in all of our supported platforms.

Given the work that I've already done, it would probably take a couple of weeks to agree on the details and implement the code (including unit tests). Maybe another week for testing on the various platforms.



was (Author: ijuma):
In order to implement this properly (as opposed to a simple test), the following steps are needed:

1. Add an optional build dependency on netty-tcnative. This library contains a fork of tomcat native that is available in Maven and is available for major platforms (Linux, OS X, Windows). It also handles extracting the platform-specific JNI code at runtime (similar to snappy-java). apr and openssl need to be installed separately.
2. Provide an implementation of `SSLEngine` based on OpenSSL. The easy option would be to add an optional dependency on `netty-handler`, which includes this. If this is not acceptable, there are some alternatives like extracting the code into a separate library or copying it into Kafka.
3. Add a way to configure the `SSLEngine` implementation (OpenSSL or JDK).
4. Change `SSLFactory` to build the appropriate `SSLEngine` based on the configuration added in `3`.
5. Potentially introduce a runtime mechanism to select `OpenSslEngine` by default if the required libraries are present (since it's much faster)
6. Potentially update `SSLTransportLayer` to handle differences in behaviour between the different `SSLEngine` implementations (the need for this depends on whether we the issues reported to Netty are fixed or not). In addition to the two issues mentioned in the description, there is also an issue related to the size of the `applicationReadBuffer`.
7. Design and implement the story around the formats for keys, certificates, key chains and certificate chains supported. OpenSSL doesn't understand the JKS format since it's Java-specific. Netty uses the `PKCS#8` format for keys and PEM format for chains when the OpenSSL engine is used.
8. Update tests to test all `SSLEngine` implementations.

Testing of this is more complicated than usual due to the native code aspect and we would have to test it in all of our supported platforms.

Given the work that I've already done, it would probably take a couple of weeks to agree on the details and implement the code (including unit tests). Maybe another week for testing on the various platforms.


> Optionally support OpenSSL for SSL/TLS 
> ---------------------------------------
>
>                 Key: KAFKA-2561
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2561
>             Project: Kafka
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.9.0.0
>            Reporter: Ismael Juma
>
> JDK's `SSLEngine` is unfortunately a bit slow (KAFKA-2431 covers this in more detail). We should consider supporting OpenSSL for SSL/TLS. Initial experiments on my laptop show that it performs a lot better:
> {code}
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, config
> 2015-09-21 14:41:58:245, 2015-09-21 14:47:02:583, 28610.2295, 94.0081, 30000000, 98574.6111, Java 8u60/server auth JDK SSLEngine/TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> 2015-09-21 14:38:24:526, 2015-09-21 14:40:19:941, 28610.2295, 247.8900, 30000000, 259931.5514, Java 8u60/server auth OpenSslEngine/TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> 2015-09-21 14:49:03:062, 2015-09-21 14:50:27:764, 28610.2295, 337.7751, 30000000, 354182.9000, Java 8u60/plaintext
> {code}
> Extracting the throughput figures:
> * JDK SSLEngine: 94 MB/s
> * OpenSSL SSLEngine: 247 MB/s
> * Plaintext: 337 MB/s (code from trunk, so no zero-copy due to KAFKA-2517)
> In order to get these figures, I used Netty's `OpenSslEngine` by hacking `SSLFactory` to use Netty's `SslContextBuilder` and made a few changes to `SSLTransportLayer` in order to workaround differences in behaviour between `OpenSslEngine` and JDK's SSLEngine (filed https://github.com/netty/netty/issues/4235 and https://github.com/netty/netty/issues/4238 upstream).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)