You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andreas Martens1 <am...@uk.ibm.com> on 2023/10/26 11:48:51 UTC
Java 1.8 and TLSv1.3
Hello good people of Kafka,
I was recently informed that TLS 1.3 doesn’t work for connecting our product to Kafka, and after some digging realised it was true, no matter how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no applicable Ciphers.
So after a bunch more digging I realised that the problem lies in the Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
static {
if (Java.IS_JAVA11_COMPATIBLE) {
DEFAULT_SSL_PROTOCOL = "TLSv1.3";
DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
} else {
DEFAULT_SSL_PROTOCOL = "TLSv1.2";
DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
}
}
My initial thoughts were that these just set the defaults, I should still be able to set TLSv1.3 in my properties, but no. If I change the above block to:
static {
DEFAULT_SSL_PROTOCOL = "TLSv1.3";
DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
}
it works just fine. I suspect (but haven’t yet had the time to verify) that there’s something that gets the list of supported Ciphers from the default, and applies those Ciphers to the selected end protocol, and since there’s no overlap I’m outta luck.
Now of course life is never simple, so I can’t just make the above change and send you fine people a PR, since there’s probably still some older JREs out there that don’t support TLSv1.3.
As a fine person once told me (actually, a Java support person) “don’t test the symptom, test the cause”. In this case, we shouldn’t be testing whether we’re working with a Java 11 JVM, we should test whether our current JVM has a TLSv1.3 Context instance. E.g.:
try{
SSLContext context = SSLContext.getInstance("TLSv1.3");
DEFAULT_SSL_PROTOCOL = "TLSv1.3";
}
catch(java.security.NoSuchAlgorithmException e){
DEFAULT_SSL_PROTOCOL = "TLSv1.2";
}
But the test in SslConfigs.java is done in *static* init, and as we all know, performing try-catch within a static is a Big No-No.
Soooo, before I go digging further in the code and start looking to modify the places where the defaults are consumed, does anyone have a better idea? My initial thought was to raise a ticket and run away, but I’m trying to be a good citizen!
I should probably mention, this code was introduced in https://issues.apache.org/jira/browse/KAFKA-7251 and https://issues.apache.org/jira/browse/KAFKA-9320 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default.
Thanks,
Andreas Martens
--
Senior Engineer
IBM App Connect Enterprise
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
RE: Java 1.8 and TLSv1.3
Posted by Andreas Martens1 <am...@uk.ibm.com>.
Ahh, that’s it, the second property needs to be set too, we weren’t doing that!
By setting both ssl.protocol and ssl.enabled.protocols we can get TLSv1.3. Annoyingly the JRE implementations differ, some interpret “TLSv1.3” to mean “TLS 1.3 and TLS 1.2” and some to mean “TLS 1.3 only”. I can’t see any way to specify, across JRE implementations, “allow both TLS 1.2 and 1.3”.
Thanks
Andreas
From: Edoardo Comar <ed...@gmail.com>
Date: Monday, 6 November 2023 at 12:43
To: users@kafka.apache.org <us...@kafka.apache.org>
Cc: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
Hi Andreas,
I just tried running a Kafka 3.6 client on a recent Temurin Jdk 8
(OpenJDK8U-jdk_x64_mac_hotspot_8u392b08) and configured it with
clientProps.put(SslConfigs.SSL_PROTOCOL_CONFIG,"TLSv1.3");
clientProps.put(SslConfigs.SSL_ENABLED_PROTOCOLS_CONFIG,"TLSv1.3");
and it connects without issues for me
On Mon, 6 Nov 2023 at 11:50, Andreas Martens1 <am...@uk.ibm.com> wrote:
>
> Hi Edoardo,
>
> Yes, that’s exactly what I’m saying!
>
> If I run the console consumer in the build with:
> kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
>
> Where consumer.properties contains:
> sasl.mechanism=PLAIN
> security.protocol=SASL_SSL
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
> username="<username>" \
> password="<password>";
>
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=changeit
> ssl.truststore.type=JKS
>
> ssl.enabled.protocols=TLSv1.3
>
> With Java 11 it works fine, but with Java 1.8 I get:
> [main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
> Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
> at com.ibm.jsse2.aa.<init>(aa.java:9)
> at com.ibm.jsse2.ab.<init>(ab.java:44)
> at com.ibm.jsse2.bb.a(bb.java:129)
> at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
> at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
> at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
> at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
> at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
>
> Thanks,
> Andreas
>
>
>
> From: Edoardo Comar <ed...@gmail.com>
> Date: Friday, 3 November 2023 at 17:26
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Cc: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
> Andreas,
> do you mean that even if you configure your Java client running on Java8 with
> ssl.enabled.protocols=TLSv1.3
> you can't connect to a Kafka broker using TLS1.3 ?
>
>
> On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> > Hi Andreas,
> >
> > The TLS code has run into changes in behavior across different Java
> > versions, so we only wanted to allow TLS 1.3 in the versions we tested
> > against. TLS 1.3 landed in Java 8 a while after we made the relevant
> > changes for Java 11 and newer. That said, Java 8 support is deprecated and
> > will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> > we want to invest in making more features available for that version.
> >
> > Thanks,
> > Ismael
> >
> > On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> > wrote:
> >
> > > Hello good people of Kafka,
> > >
> > > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > > product to Kafka, and after some digging realised it was true, no matter
> > > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > > applicable Ciphers.
> > >
> > > So after a bunch more digging I realised that the problem lies in the
> > > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > > static {
> > > if (Java.IS_JAVA11_COMPATIBLE) {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > } else {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > > }
> > > }
> > >
> > > My initial thoughts were that these just set the defaults, I should still
> > > be able to set TLSv1.3 in my properties, but no. If I change the above
> > > block to:
> > > static {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > }
> > > it works just fine. I suspect (but haven’t yet had the time to verify)
> > > that there’s something that gets the list of supported Ciphers from the
> > > default, and applies those Ciphers to the selected end protocol, and since
> > > there’s no overlap I’m outta luck.
> > >
> > > Now of course life is never simple, so I can’t just make the above change
> > > and send you fine people a PR, since there’s probably still some older JREs
> > > out there that don’t support TLSv1.3.
> > >
> > > As a fine person once told me (actually, a Java support person) “don’t
> > > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > > whether we’re working with a Java 11 JVM, we should test whether our
> > > current JVM has a TLSv1.3 Context instance. E.g.:
> > > try{
> > > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > }
> > > catch(java.security.NoSuchAlgorithmException e){
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > }
> > > But the test in SslConfigs.java is done in *static* init, and as we all
> > > know, performing try-catch within a static is a Big No-No.
> > >
> > > Soooo, before I go digging further in the code and start looking to modify
> > > the places where the defaults are consumed, does anyone have a better idea?
> > > My initial thought was to raise a ticket and run away, but I’m trying to be
> > > a good citizen!
> > >
> > > I should probably mention, this code was introduced in
> > > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > > .
> > >
> > > Thanks,
> > > Andreas Martens
> > > --
> > > Senior Engineer
> > > IBM App Connect Enterprise
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
RE: Java 1.8 and TLSv1.3
Posted by Andreas Martens1 <am...@uk.ibm.com>.
Ahh, that’s it, the second property needs to be set too, we weren’t doing that!
By setting both ssl.protocol and ssl.enabled.protocols we can get TLSv1.3. Annoyingly the JRE implementations differ, some interpret “TLSv1.3” to mean “TLS 1.3 and TLS 1.2” and some to mean “TLS 1.3 only”. I can’t see any way to specify, across JRE implementations, “allow both TLS 1.2 and 1.3”.
Thanks
Andreas
From: Edoardo Comar <ed...@gmail.com>
Date: Monday, 6 November 2023 at 12:43
To: users@kafka.apache.org <us...@kafka.apache.org>
Cc: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
Hi Andreas,
I just tried running a Kafka 3.6 client on a recent Temurin Jdk 8
(OpenJDK8U-jdk_x64_mac_hotspot_8u392b08) and configured it with
clientProps.put(SslConfigs.SSL_PROTOCOL_CONFIG,"TLSv1.3");
clientProps.put(SslConfigs.SSL_ENABLED_PROTOCOLS_CONFIG,"TLSv1.3");
and it connects without issues for me
On Mon, 6 Nov 2023 at 11:50, Andreas Martens1 <am...@uk.ibm.com> wrote:
>
> Hi Edoardo,
>
> Yes, that’s exactly what I’m saying!
>
> If I run the console consumer in the build with:
> kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
>
> Where consumer.properties contains:
> sasl.mechanism=PLAIN
> security.protocol=SASL_SSL
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
> username="<username>" \
> password="<password>";
>
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=changeit
> ssl.truststore.type=JKS
>
> ssl.enabled.protocols=TLSv1.3
>
> With Java 11 it works fine, but with Java 1.8 I get:
> [main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
> Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
> at com.ibm.jsse2.aa.<init>(aa.java:9)
> at com.ibm.jsse2.ab.<init>(ab.java:44)
> at com.ibm.jsse2.bb.a(bb.java:129)
> at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
> at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
> at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
> at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
> at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
>
> Thanks,
> Andreas
>
>
>
> From: Edoardo Comar <ed...@gmail.com>
> Date: Friday, 3 November 2023 at 17:26
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Cc: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
> Andreas,
> do you mean that even if you configure your Java client running on Java8 with
> ssl.enabled.protocols=TLSv1.3
> you can't connect to a Kafka broker using TLS1.3 ?
>
>
> On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> > Hi Andreas,
> >
> > The TLS code has run into changes in behavior across different Java
> > versions, so we only wanted to allow TLS 1.3 in the versions we tested
> > against. TLS 1.3 landed in Java 8 a while after we made the relevant
> > changes for Java 11 and newer. That said, Java 8 support is deprecated and
> > will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> > we want to invest in making more features available for that version.
> >
> > Thanks,
> > Ismael
> >
> > On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> > wrote:
> >
> > > Hello good people of Kafka,
> > >
> > > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > > product to Kafka, and after some digging realised it was true, no matter
> > > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > > applicable Ciphers.
> > >
> > > So after a bunch more digging I realised that the problem lies in the
> > > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > > static {
> > > if (Java.IS_JAVA11_COMPATIBLE) {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > } else {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > > }
> > > }
> > >
> > > My initial thoughts were that these just set the defaults, I should still
> > > be able to set TLSv1.3 in my properties, but no. If I change the above
> > > block to:
> > > static {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > }
> > > it works just fine. I suspect (but haven’t yet had the time to verify)
> > > that there’s something that gets the list of supported Ciphers from the
> > > default, and applies those Ciphers to the selected end protocol, and since
> > > there’s no overlap I’m outta luck.
> > >
> > > Now of course life is never simple, so I can’t just make the above change
> > > and send you fine people a PR, since there’s probably still some older JREs
> > > out there that don’t support TLSv1.3.
> > >
> > > As a fine person once told me (actually, a Java support person) “don’t
> > > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > > whether we’re working with a Java 11 JVM, we should test whether our
> > > current JVM has a TLSv1.3 Context instance. E.g.:
> > > try{
> > > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > }
> > > catch(java.security.NoSuchAlgorithmException e){
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > }
> > > But the test in SslConfigs.java is done in *static* init, and as we all
> > > know, performing try-catch within a static is a Big No-No.
> > >
> > > Soooo, before I go digging further in the code and start looking to modify
> > > the places where the defaults are consumed, does anyone have a better idea?
> > > My initial thought was to raise a ticket and run away, but I’m trying to be
> > > a good citizen!
> > >
> > > I should probably mention, this code was introduced in
> > > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > > .
> > >
> > > Thanks,
> > > Andreas Martens
> > > --
> > > Senior Engineer
> > > IBM App Connect Enterprise
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Re: Java 1.8 and TLSv1.3
Posted by Edoardo Comar <ed...@gmail.com>.
Hi Andreas,
I just tried running a Kafka 3.6 client on a recent Temurin Jdk 8
(OpenJDK8U-jdk_x64_mac_hotspot_8u392b08) and configured it with
clientProps.put(SslConfigs.SSL_PROTOCOL_CONFIG,"TLSv1.3");
clientProps.put(SslConfigs.SSL_ENABLED_PROTOCOLS_CONFIG,"TLSv1.3");
and it connects without issues for me
On Mon, 6 Nov 2023 at 11:50, Andreas Martens1 <am...@uk.ibm.com> wrote:
>
> Hi Edoardo,
>
> Yes, that’s exactly what I’m saying!
>
> If I run the console consumer in the build with:
> kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
>
> Where consumer.properties contains:
> sasl.mechanism=PLAIN
> security.protocol=SASL_SSL
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
> username="<username>" \
> password="<password>";
>
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=changeit
> ssl.truststore.type=JKS
>
> ssl.enabled.protocols=TLSv1.3
>
> With Java 11 it works fine, but with Java 1.8 I get:
> [main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
> Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
> at com.ibm.jsse2.aa.<init>(aa.java:9)
> at com.ibm.jsse2.ab.<init>(ab.java:44)
> at com.ibm.jsse2.bb.a(bb.java:129)
> at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
> at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
> at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
> at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
> at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
>
> Thanks,
> Andreas
>
>
>
> From: Edoardo Comar <ed...@gmail.com>
> Date: Friday, 3 November 2023 at 17:26
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Cc: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
> Andreas,
> do you mean that even if you configure your Java client running on Java8 with
> ssl.enabled.protocols=TLSv1.3
> you can't connect to a Kafka broker using TLS1.3 ?
>
>
> On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> > Hi Andreas,
> >
> > The TLS code has run into changes in behavior across different Java
> > versions, so we only wanted to allow TLS 1.3 in the versions we tested
> > against. TLS 1.3 landed in Java 8 a while after we made the relevant
> > changes for Java 11 and newer. That said, Java 8 support is deprecated and
> > will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> > we want to invest in making more features available for that version.
> >
> > Thanks,
> > Ismael
> >
> > On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> > wrote:
> >
> > > Hello good people of Kafka,
> > >
> > > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > > product to Kafka, and after some digging realised it was true, no matter
> > > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > > applicable Ciphers.
> > >
> > > So after a bunch more digging I realised that the problem lies in the
> > > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > > static {
> > > if (Java.IS_JAVA11_COMPATIBLE) {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > } else {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > > }
> > > }
> > >
> > > My initial thoughts were that these just set the defaults, I should still
> > > be able to set TLSv1.3 in my properties, but no. If I change the above
> > > block to:
> > > static {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > }
> > > it works just fine. I suspect (but haven’t yet had the time to verify)
> > > that there’s something that gets the list of supported Ciphers from the
> > > default, and applies those Ciphers to the selected end protocol, and since
> > > there’s no overlap I’m outta luck.
> > >
> > > Now of course life is never simple, so I can’t just make the above change
> > > and send you fine people a PR, since there’s probably still some older JREs
> > > out there that don’t support TLSv1.3.
> > >
> > > As a fine person once told me (actually, a Java support person) “don’t
> > > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > > whether we’re working with a Java 11 JVM, we should test whether our
> > > current JVM has a TLSv1.3 Context instance. E.g.:
> > > try{
> > > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > }
> > > catch(java.security.NoSuchAlgorithmException e){
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > }
> > > But the test in SslConfigs.java is done in *static* init, and as we all
> > > know, performing try-catch within a static is a Big No-No.
> > >
> > > Soooo, before I go digging further in the code and start looking to modify
> > > the places where the defaults are consumed, does anyone have a better idea?
> > > My initial thought was to raise a ticket and run away, but I’m trying to be
> > > a good citizen!
> > >
> > > I should probably mention, this code was introduced in
> > > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > > .
> > >
> > > Thanks,
> > > Andreas Martens
> > > --
> > > Senior Engineer
> > > IBM App Connect Enterprise
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Re: Java 1.8 and TLSv1.3
Posted by Edoardo Comar <ed...@gmail.com>.
Hi Andreas,
I just tried running a Kafka 3.6 client on a recent Temurin Jdk 8
(OpenJDK8U-jdk_x64_mac_hotspot_8u392b08) and configured it with
clientProps.put(SslConfigs.SSL_PROTOCOL_CONFIG,"TLSv1.3");
clientProps.put(SslConfigs.SSL_ENABLED_PROTOCOLS_CONFIG,"TLSv1.3");
and it connects without issues for me
On Mon, 6 Nov 2023 at 11:50, Andreas Martens1 <am...@uk.ibm.com> wrote:
>
> Hi Edoardo,
>
> Yes, that’s exactly what I’m saying!
>
> If I run the console consumer in the build with:
> kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
>
> Where consumer.properties contains:
> sasl.mechanism=PLAIN
> security.protocol=SASL_SSL
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
> username="<username>" \
> password="<password>";
>
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=changeit
> ssl.truststore.type=JKS
>
> ssl.enabled.protocols=TLSv1.3
>
> With Java 11 it works fine, but with Java 1.8 I get:
> [main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
> Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
> at com.ibm.jsse2.aa.<init>(aa.java:9)
> at com.ibm.jsse2.ab.<init>(ab.java:44)
> at com.ibm.jsse2.bb.a(bb.java:129)
> at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
> at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
> at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
> at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
> at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
> at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
>
> Thanks,
> Andreas
>
>
>
> From: Edoardo Comar <ed...@gmail.com>
> Date: Friday, 3 November 2023 at 17:26
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Cc: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
> Andreas,
> do you mean that even if you configure your Java client running on Java8 with
> ssl.enabled.protocols=TLSv1.3
> you can't connect to a Kafka broker using TLS1.3 ?
>
>
> On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> > Hi Andreas,
> >
> > The TLS code has run into changes in behavior across different Java
> > versions, so we only wanted to allow TLS 1.3 in the versions we tested
> > against. TLS 1.3 landed in Java 8 a while after we made the relevant
> > changes for Java 11 and newer. That said, Java 8 support is deprecated and
> > will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> > we want to invest in making more features available for that version.
> >
> > Thanks,
> > Ismael
> >
> > On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> > wrote:
> >
> > > Hello good people of Kafka,
> > >
> > > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > > product to Kafka, and after some digging realised it was true, no matter
> > > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > > applicable Ciphers.
> > >
> > > So after a bunch more digging I realised that the problem lies in the
> > > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > > static {
> > > if (Java.IS_JAVA11_COMPATIBLE) {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > } else {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > > }
> > > }
> > >
> > > My initial thoughts were that these just set the defaults, I should still
> > > be able to set TLSv1.3 in my properties, but no. If I change the above
> > > block to:
> > > static {
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > > }
> > > it works just fine. I suspect (but haven’t yet had the time to verify)
> > > that there’s something that gets the list of supported Ciphers from the
> > > default, and applies those Ciphers to the selected end protocol, and since
> > > there’s no overlap I’m outta luck.
> > >
> > > Now of course life is never simple, so I can’t just make the above change
> > > and send you fine people a PR, since there’s probably still some older JREs
> > > out there that don’t support TLSv1.3.
> > >
> > > As a fine person once told me (actually, a Java support person) “don’t
> > > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > > whether we’re working with a Java 11 JVM, we should test whether our
> > > current JVM has a TLSv1.3 Context instance. E.g.:
> > > try{
> > > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > > }
> > > catch(java.security.NoSuchAlgorithmException e){
> > > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > > }
> > > But the test in SslConfigs.java is done in *static* init, and as we all
> > > know, performing try-catch within a static is a Big No-No.
> > >
> > > Soooo, before I go digging further in the code and start looking to modify
> > > the places where the defaults are consumed, does anyone have a better idea?
> > > My initial thought was to raise a ticket and run away, but I’m trying to be
> > > a good citizen!
> > >
> > > I should probably mention, this code was introduced in
> > > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > > .
> > >
> > > Thanks,
> > > Andreas Martens
> > > --
> > > Senior Engineer
> > > IBM App Connect Enterprise
> > >
> > > Unless otherwise stated above:
> > >
> > > IBM United Kingdom Limited
> > > Registered in England and Wales with number 741598
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> > >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
RE: Java 1.8 and TLSv1.3
Posted by Andreas Martens1 <am...@uk.ibm.com>.
Hi Edoardo,
Yes, that’s exactly what I’m saying!
If I run the console consumer in the build with:
kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
Where consumer.properties contains:
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<username>" \
password="<password>";
ssl.truststore.location=truststore.jks
ssl.truststore.password=changeit
ssl.truststore.type=JKS
ssl.enabled.protocols=TLSv1.3
With Java 11 it works fine, but with Java 1.8 I get:
[main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at com.ibm.jsse2.aa.<init>(aa.java:9)
at com.ibm.jsse2.ab.<init>(ab.java:44)
at com.ibm.jsse2.bb.a(bb.java:129)
at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
Thanks,
Andreas
From: Edoardo Comar <ed...@gmail.com>
Date: Friday, 3 November 2023 at 17:26
To: users@kafka.apache.org <us...@kafka.apache.org>
Cc: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
Andreas,
do you mean that even if you configure your Java client running on Java8 with
ssl.enabled.protocols=TLSv1.3
you can't connect to a Kafka broker using TLS1.3 ?
On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
>
> Hi Andreas,
>
> The TLS code has run into changes in behavior across different Java
> versions, so we only wanted to allow TLS 1.3 in the versions we tested
> against. TLS 1.3 landed in Java 8 a while after we made the relevant
> changes for Java 11 and newer. That said, Java 8 support is deprecated and
> will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> we want to invest in making more features available for that version.
>
> Thanks,
> Ismael
>
> On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> wrote:
>
> > Hello good people of Kafka,
> >
> > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > product to Kafka, and after some digging realised it was true, no matter
> > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > applicable Ciphers.
> >
> > So after a bunch more digging I realised that the problem lies in the
> > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > static {
> > if (Java.IS_JAVA11_COMPATIBLE) {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > } else {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > }
> > }
> >
> > My initial thoughts were that these just set the defaults, I should still
> > be able to set TLSv1.3 in my properties, but no. If I change the above
> > block to:
> > static {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > }
> > it works just fine. I suspect (but haven’t yet had the time to verify)
> > that there’s something that gets the list of supported Ciphers from the
> > default, and applies those Ciphers to the selected end protocol, and since
> > there’s no overlap I’m outta luck.
> >
> > Now of course life is never simple, so I can’t just make the above change
> > and send you fine people a PR, since there’s probably still some older JREs
> > out there that don’t support TLSv1.3.
> >
> > As a fine person once told me (actually, a Java support person) “don’t
> > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > whether we’re working with a Java 11 JVM, we should test whether our
> > current JVM has a TLSv1.3 Context instance. E.g.:
> > try{
> > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > }
> > catch(java.security.NoSuchAlgorithmException e){
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > }
> > But the test in SslConfigs.java is done in *static* init, and as we all
> > know, performing try-catch within a static is a Big No-No.
> >
> > Soooo, before I go digging further in the code and start looking to modify
> > the places where the defaults are consumed, does anyone have a better idea?
> > My initial thought was to raise a ticket and run away, but I’m trying to be
> > a good citizen!
> >
> > I should probably mention, this code was introduced in
> > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > .
> >
> > Thanks,
> > Andreas Martens
> > --
> > Senior Engineer
> > IBM App Connect Enterprise
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
RE: Java 1.8 and TLSv1.3
Posted by Andreas Martens1 <am...@uk.ibm.com>.
Hi Edoardo,
Yes, that’s exactly what I’m saying!
If I run the console consumer in the build with:
kafka-console-consumer.sh --bootstrap-server <server>:443 --topic <topic> --consumer.config $PWD/consumer.properties
Where consumer.properties contains:
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<username>" \
password="<password>";
ssl.truststore.location=truststore.jks
ssl.truststore.password=changeit
ssl.truststore.type=JKS
ssl.enabled.protocols=TLSv1.3
With Java 11 it works fine, but with Java 1.8 I get:
[main] ERROR kafka.tools.ConsoleConsumer$ - Error processing message, terminating consumer process:
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at com.ibm.jsse2.aa.<init>(aa.java:9)
at com.ibm.jsse2.ab.<init>(ab.java:44)
at com.ibm.jsse2.bb.a(bb.java:129)
at com.ibm.jsse2.bg.beginHandshake(bg.java:141)
at org.apache.kafka.common.network.SslTransportLayer.startHandshake(SslTransportLayer.java:125)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:231)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:274)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:248)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.coordinatorUnknownAndUnreadySync(ConsumerCoordinator.java:443)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:475)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1296)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1255)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1235)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:473)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
If I change the test in SslConfigs.java to test for presence of the TPSv1.3 protocol rather than testing for Java version, then both versions connect OK.
Thanks,
Andreas
From: Edoardo Comar <ed...@gmail.com>
Date: Friday, 3 November 2023 at 17:26
To: users@kafka.apache.org <us...@kafka.apache.org>
Cc: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: [EXTERNAL] Re: Java 1.8 and TLSv1.3
Andreas,
do you mean that even if you configure your Java client running on Java8 with
ssl.enabled.protocols=TLSv1.3
you can't connect to a Kafka broker using TLS1.3 ?
On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
>
> Hi Andreas,
>
> The TLS code has run into changes in behavior across different Java
> versions, so we only wanted to allow TLS 1.3 in the versions we tested
> against. TLS 1.3 landed in Java 8 a while after we made the relevant
> changes for Java 11 and newer. That said, Java 8 support is deprecated and
> will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> we want to invest in making more features available for that version.
>
> Thanks,
> Ismael
>
> On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> wrote:
>
> > Hello good people of Kafka,
> >
> > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > product to Kafka, and after some digging realised it was true, no matter
> > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > applicable Ciphers.
> >
> > So after a bunch more digging I realised that the problem lies in the
> > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > static {
> > if (Java.IS_JAVA11_COMPATIBLE) {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > } else {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > }
> > }
> >
> > My initial thoughts were that these just set the defaults, I should still
> > be able to set TLSv1.3 in my properties, but no. If I change the above
> > block to:
> > static {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > }
> > it works just fine. I suspect (but haven’t yet had the time to verify)
> > that there’s something that gets the list of supported Ciphers from the
> > default, and applies those Ciphers to the selected end protocol, and since
> > there’s no overlap I’m outta luck.
> >
> > Now of course life is never simple, so I can’t just make the above change
> > and send you fine people a PR, since there’s probably still some older JREs
> > out there that don’t support TLSv1.3.
> >
> > As a fine person once told me (actually, a Java support person) “don’t
> > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > whether we’re working with a Java 11 JVM, we should test whether our
> > current JVM has a TLSv1.3 Context instance. E.g.:
> > try{
> > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > }
> > catch(java.security.NoSuchAlgorithmException e){
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > }
> > But the test in SslConfigs.java is done in *static* init, and as we all
> > know, performing try-catch within a static is a Big No-No.
> >
> > Soooo, before I go digging further in the code and start looking to modify
> > the places where the defaults are consumed, does anyone have a better idea?
> > My initial thought was to raise a ticket and run away, but I’m trying to be
> > a good citizen!
> >
> > I should probably mention, this code was introduced in
> > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > .
> >
> > Thanks,
> > Andreas Martens
> > --
> > Senior Engineer
> > IBM App Connect Enterprise
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
Re: Java 1.8 and TLSv1.3
Posted by Edoardo Comar <ed...@gmail.com>.
Andreas,
do you mean that even if you configure your Java client running on Java8 with
ssl.enabled.protocols=TLSv1.3
you can't connect to a Kafka broker using TLS1.3 ?
On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
>
> Hi Andreas,
>
> The TLS code has run into changes in behavior across different Java
> versions, so we only wanted to allow TLS 1.3 in the versions we tested
> against. TLS 1.3 landed in Java 8 a while after we made the relevant
> changes for Java 11 and newer. That said, Java 8 support is deprecated and
> will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> we want to invest in making more features available for that version.
>
> Thanks,
> Ismael
>
> On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> wrote:
>
> > Hello good people of Kafka,
> >
> > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > product to Kafka, and after some digging realised it was true, no matter
> > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > applicable Ciphers.
> >
> > So after a bunch more digging I realised that the problem lies in the
> > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > static {
> > if (Java.IS_JAVA11_COMPATIBLE) {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > } else {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > }
> > }
> >
> > My initial thoughts were that these just set the defaults, I should still
> > be able to set TLSv1.3 in my properties, but no. If I change the above
> > block to:
> > static {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > }
> > it works just fine. I suspect (but haven’t yet had the time to verify)
> > that there’s something that gets the list of supported Ciphers from the
> > default, and applies those Ciphers to the selected end protocol, and since
> > there’s no overlap I’m outta luck.
> >
> > Now of course life is never simple, so I can’t just make the above change
> > and send you fine people a PR, since there’s probably still some older JREs
> > out there that don’t support TLSv1.3.
> >
> > As a fine person once told me (actually, a Java support person) “don’t
> > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > whether we’re working with a Java 11 JVM, we should test whether our
> > current JVM has a TLSv1.3 Context instance. E.g.:
> > try{
> > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > }
> > catch(java.security.NoSuchAlgorithmException e){
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > }
> > But the test in SslConfigs.java is done in *static* init, and as we all
> > know, performing try-catch within a static is a Big No-No.
> >
> > Soooo, before I go digging further in the code and start looking to modify
> > the places where the defaults are consumed, does anyone have a better idea?
> > My initial thought was to raise a ticket and run away, but I’m trying to be
> > a good citizen!
> >
> > I should probably mention, this code was introduced in
> > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > .
> >
> > Thanks,
> > Andreas Martens
> > --
> > Senior Engineer
> > IBM App Connect Enterprise
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
Re: Java 1.8 and TLSv1.3
Posted by Edoardo Comar <ed...@gmail.com>.
Andreas,
do you mean that even if you configure your Java client running on Java8 with
ssl.enabled.protocols=TLSv1.3
you can't connect to a Kafka broker using TLS1.3 ?
On Sat, 28 Oct 2023 at 01:03, Ismael Juma <me...@ismaeljuma.com> wrote:
>
> Hi Andreas,
>
> The TLS code has run into changes in behavior across different Java
> versions, so we only wanted to allow TLS 1.3 in the versions we tested
> against. TLS 1.3 landed in Java 8 a while after we made the relevant
> changes for Java 11 and newer. That said, Java 8 support is deprecated and
> will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
> we want to invest in making more features available for that version.
>
> Thanks,
> Ismael
>
> On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
> wrote:
>
> > Hello good people of Kafka,
> >
> > I was recently informed that TLS 1.3 doesn’t work for connecting our
> > product to Kafka, and after some digging realised it was true, no matter
> > how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> > applicable Ciphers.
> >
> > So after a bunch more digging I realised that the problem lies in the
> > Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> > static {
> > if (Java.IS_JAVA11_COMPATIBLE) {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > } else {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> > }
> > }
> >
> > My initial thoughts were that these just set the defaults, I should still
> > be able to set TLSv1.3 in my properties, but no. If I change the above
> > block to:
> > static {
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> > }
> > it works just fine. I suspect (but haven’t yet had the time to verify)
> > that there’s something that gets the list of supported Ciphers from the
> > default, and applies those Ciphers to the selected end protocol, and since
> > there’s no overlap I’m outta luck.
> >
> > Now of course life is never simple, so I can’t just make the above change
> > and send you fine people a PR, since there’s probably still some older JREs
> > out there that don’t support TLSv1.3.
> >
> > As a fine person once told me (actually, a Java support person) “don’t
> > test the symptom, test the cause”. In this case, we shouldn’t be testing
> > whether we’re working with a Java 11 JVM, we should test whether our
> > current JVM has a TLSv1.3 Context instance. E.g.:
> > try{
> > SSLContext context = SSLContext.getInstance("TLSv1.3");
> > DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> > }
> > catch(java.security.NoSuchAlgorithmException e){
> > DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> > }
> > But the test in SslConfigs.java is done in *static* init, and as we all
> > know, performing try-catch within a static is a Big No-No.
> >
> > Soooo, before I go digging further in the code and start looking to modify
> > the places where the defaults are consumed, does anyone have a better idea?
> > My initial thought was to raise a ticket and run away, but I’m trying to be
> > a good citizen!
> >
> > I should probably mention, this code was introduced in
> > https://issues.apache.org/jira/browse/KAFKA-7251 and
> > https://issues.apache.org/jira/browse/KAFKA-9320 and
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> > .
> >
> > Thanks,
> > Andreas Martens
> > --
> > Senior Engineer
> > IBM App Connect Enterprise
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
Re: Java 1.8 and TLSv1.3
Posted by Ismael Juma <me...@ismaeljuma.com>.
Hi Andreas,
The TLS code has run into changes in behavior across different Java
versions, so we only wanted to allow TLS 1.3 in the versions we tested
against. TLS 1.3 landed in Java 8 a while after we made the relevant
changes for Java 11 and newer. That said, Java 8 support is deprecated and
will be removed in Apache Kafka 4.0 (in a few months). So, it's not clear
we want to invest in making more features available for that version.
Thanks,
Ismael
On Thu, Oct 26, 2023 at 4:49 AM Andreas Martens1 <am...@uk.ibm.com>
wrote:
> Hello good people of Kafka,
>
> I was recently informed that TLS 1.3 doesn’t work for connecting our
> product to Kafka, and after some digging realised it was true, no matter
> how hard I type “TLSv1.3” it doesn’t work, weirdly with an error about no
> applicable Ciphers.
>
> So after a bunch more digging I realised that the problem lies in the
> Kafka client classes, in Kafka clients’ SslConfigs.java there is this code:
> static {
> if (Java.IS_JAVA11_COMPATIBLE) {
> DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> } else {
> DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2";
> }
> }
>
> My initial thoughts were that these just set the defaults, I should still
> be able to set TLSv1.3 in my properties, but no. If I change the above
> block to:
> static {
> DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> DEFAULT_SSL_ENABLED_PROTOCOLS = "TLSv1.2,TLSv1.3";
> }
> it works just fine. I suspect (but haven’t yet had the time to verify)
> that there’s something that gets the list of supported Ciphers from the
> default, and applies those Ciphers to the selected end protocol, and since
> there’s no overlap I’m outta luck.
>
> Now of course life is never simple, so I can’t just make the above change
> and send you fine people a PR, since there’s probably still some older JREs
> out there that don’t support TLSv1.3.
>
> As a fine person once told me (actually, a Java support person) “don’t
> test the symptom, test the cause”. In this case, we shouldn’t be testing
> whether we’re working with a Java 11 JVM, we should test whether our
> current JVM has a TLSv1.3 Context instance. E.g.:
> try{
> SSLContext context = SSLContext.getInstance("TLSv1.3");
> DEFAULT_SSL_PROTOCOL = "TLSv1.3";
> }
> catch(java.security.NoSuchAlgorithmException e){
> DEFAULT_SSL_PROTOCOL = "TLSv1.2";
> }
> But the test in SslConfigs.java is done in *static* init, and as we all
> know, performing try-catch within a static is a Big No-No.
>
> Soooo, before I go digging further in the code and start looking to modify
> the places where the defaults are consumed, does anyone have a better idea?
> My initial thought was to raise a ticket and run away, but I’m trying to be
> a good citizen!
>
> I should probably mention, this code was introduced in
> https://issues.apache.org/jira/browse/KAFKA-7251 and
> https://issues.apache.org/jira/browse/KAFKA-9320 and
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> .
>
> Thanks,
> Andreas Martens
> --
> Senior Engineer
> IBM App Connect Enterprise
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>