You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Andy Feller (JIRA)" <ji...@apache.org> on 2018/03/05 14:09:00 UTC
[jira] [Commented] (KAFKA-2610) Metrics for SSL handshake
[ https://issues.apache.org/jira/browse/KAFKA-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386115#comment-16386115 ]
Andy Feller commented on KAFKA-2610:
------------------------------------
Will bump thread as this is beyond nice to have as I was involved in a 3 week endeavor with hundreds of consumers and producers whose SSL certificates expired in waves caused a Kafka cluster to effectively fall over and there was no way from Kafka side to easily identify issue and run down offenders. It eventually required enabling debug level logging for the following classes to determine the issue with the following stack trace:
System information:
* Apache Kafka 2.10-0.10.0.1
* Ubuntu 14 LTS
Log4j changes:
* org.apache.kafka.common.network.SslTransportLayer
* org.apache.kafka.common.network.Selector
Kafka debug stack trace to identify expired certificates:
{noformat}
[2018-02-12 15:21:29,675] DEBUG Connection with ###.##.##.##/###.##.##.## disconnected (org.apache.kafka.common.network.Selector)
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:378)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:243)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:62)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:318)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at kafka.network.Processor.poll(SocketServer.scala:472)
at kafka.network.Processor.run(SocketServer.scala:412)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1979)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:336)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:414)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:270)
... 6 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279)
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130)
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1966)
... 15 more
Caused by: java.security.cert.CertPathValidatorException: validity check failed
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
... 21 more
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun Jan 28 06:08:00 EST 2018
at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274)
at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629)
at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190)
at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
... 26 more{noformat}
Additionally, there was another stack trace being logged around SSL worth mentioning
{noformat}
[2018-02-12 15:21:29,660] DEBUG SSLEngine.closeInBound() raised an exception. (org.apache.kafka.common.network.SslTransportLayer)
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561)
at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:724)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:314)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:62)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:318)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at kafka.network.Processor.poll(SocketServer.scala:472)
at kafka.network.Processor.run(SocketServer.scala:412)
at java.lang.Thread.run(Thread.java:748){noformat}
It would be nice to have some other kind of pressure valve on top of metrics to prevent abuse, which I'm open to suggestions.
> Metrics for SSL handshake
> -------------------------
>
> Key: KAFKA-2610
> URL: https://issues.apache.org/jira/browse/KAFKA-2610
> Project: Kafka
> Issue Type: Sub-task
> Components: security
> Reporter: Ismael Juma
> Priority: Major
>
> It would be useful to report some metrics on SSL handshakes to make sure that they are taking a reasonable amount of time and that they are only happening when we expect them to happen. As part of this ticket, we should define the metrics to be implemented and implement them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)