[DISCUSS] Brokers disconnect intermittently with TLS1.3

["Kokoori, Shylaja" <sh...@intel.com>]

Hi all,

Using TLS1.3 (with JDK11) is causing an intermittent increase in inter-broker p99 latency, as mentioned by Yiming in Kafka-9320<https://issues.apache.org/jira/browse/KAFKA-9320?focusedCommentId=17401818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17401818>. We tested this with Kafka 2.8.
The issue seems to be because of a renegotiation exception being thrown by

read(ByteBuffer dst)

&

write(ByteBuffer src)

in
clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java

This exception is causing the connection to close between the brokers before read/write is completed.

In our internal experiments we have seen the p99 latency stabilize when we remove this exception.

Given that TLS1.3 does not support renegotiation, can I make it applicable just for TLS1.2?

I have also created a ticket<https://issues.apache.org/jira/browse/KAFKA-13418>

Any feedback is welcome.

Thank you,

Shylaja

Re: [DISCUSS] Brokers disconnect intermittently with TLS1.3

[Luke Chen <sh...@gmail.com>]

Hi Shylaja,
Thanks for reporting the issue.
> Given that TLS1.3 does not support renegotiation, can I make it
applicable just for TLS1.2?
Are you saying you're trying to make Kafka default supports to TLS1.2,
instead of TLS1.3?
If so, I don't think it's a good idea to fall back to an older and weaker
security protocol just because of a bug.
Instead, I think we should try to investigate it and fix it from the root.

So, are you sure this is a issue that `renegotiation` is not supported by
TLSv1.3?
Could we fix it?

Thank you.
Luke

On Tue, Nov 16, 2021 at 4:05 AM Kokoori, Shylaja <sh...@intel.com>
wrote:

> Hi all,
>
> Using TLS1.3 (with JDK11) is causing an intermittent increase in
> inter-broker p99 latency, as mentioned by Yiming in Kafka-9320<
> https://issues.apache.org/jira/browse/KAFKA-9320?focusedCommentId=17401818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17401818>.
> We tested this with Kafka 2.8.
> The issue seems to be because of a renegotiation exception being thrown by
>
> read(ByteBuffer dst)
>
> &
>
> write(ByteBuffer src)
>
> in
>
> clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java
>
> This exception is causing the connection to close between the brokers
> before read/write is completed.
>
> In our internal experiments we have seen the p99 latency stabilize when we
> remove this exception.
>
> Given that TLS1.3 does not support renegotiation, can I make it applicable
> just for TLS1.2?
>
> I have also created a ticket<
> https://issues.apache.org/jira/browse/KAFKA-13418>
>
> Any feedback is welcome.
>
> Thank you,
>
> Shylaja
>
>
>
>