You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2020/11/05 22:49:00 UTC
[jira] [Updated] (CASSANDRA-16144) TLS connections to the storage
port on a node without server encryption configured causes
java.io.IOException accessing missing keystore
[ https://issues.apache.org/jira/browse/CASSANDRA-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Capwell updated CASSANDRA-16144:
--------------------------------------
Status: Ready to Commit (was: Changes Suggested)
> TLS connections to the storage port on a node without server encryption configured causes java.io.IOException accessing missing keystore
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16144
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16144
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: Jon Meredith
> Assignee: Jon Meredith
> Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> If a TLS connection is requested against a node with all encryption disabled by configuration,
> configured with
> {code}
> server_encryption_options: {optional:false, internode_encryption: none}
> {code}
> it logs the following error if no keystore exists for the node.
> {code}
> INFO [Messaging-EventLoop-3-3] 2020-09-15T14:30:02,952 : - 127.0.0.1:7000->127.0.1.1:7000-URGENT_MESSAGES-[no-channel] failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: local1-i1/127.0.1.1:7000
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
> at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> WARN [Messaging-EventLoop-3-9] 2020-09-15T14:30:06,375 : - Failed to initialize a channel. Closing: [id: 0x0746c157, L:/127.0.0.1:7000 - R:/127.0.0.1:59623]
> java.io.IOException: failed to build trust manager store for secure connections
> at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:232) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:300) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:276) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:257) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at org.apache.cassandra.net.InboundConnectionInitiator$Initializer.initChannel(InboundConnectionInitiator.java:107) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at org.apache.cassandra.net.InboundConnectionInitiator$Initializer.initChannel(InboundConnectionInitiator.java:71) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.50.Final.jar:4.1.50.Final]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: java.nio.file.NoSuchFileException: conf/.keystore
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
> at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
> at java.nio.file.Files.newByteChannel(Files.java:370) ~[?:?]
> at java.nio.file.Files.newByteChannel(Files.java:421) ~[?:?]
> at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
> at java.nio.file.Files.newInputStream(Files.java:155) ~[?:?]
> at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:207) ~[apache-cassandra-4.0-beta1-SNAPSHOT.jar:4.0-beta1-SNAPSHOT]
> ... 23 more
> {code}
> This happens regardless of the settings of server_encryption_options.optional, as the OptionalSslHandler is installed if optional is true, and if optional is not true, an SSL handler is always installed.
> https://github.com/apache/cassandra/blob/674b6cc1a5e905a9c234c649adaad2de79cfa560/src/java/org/apache/cassandra/net/InboundConnectionInitiator.java#L101
> CASSANDRA-15262 improved backward 4.0 backward compatability with 3.11 by defaulting to allowing optional encrypted connections if optional was not explicitly configured, however if an operator has not installed a keystore and an secure connection cannot possibly be established, neither the optional or required SSL handlers should be installed.
> Similarly if the operator has explicitly disabled SSL, then neither SSL handlers should be installed.
> Temporarily commenting out both handlers and testing a TLS connection logs, an incoming TLS connection logs an ERROR
> {code}
> ERROR [Messaging-EventLoop-3-1] 2020-09-15 15:09:48,898 InboundConnectionInitiator.java:355 - Failed to properly handshake with peer /127.0.0.1:60525. Closing the channel.
> io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 369295616, Expected -900387334
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 369295616, Expected -900387334
> at org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:333)
> at org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
> at org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:255)
> at org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:248)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
> ... 17 common frames omitted
> {code}
> I think it would be helpful to an include a {{DisabledSslHandler}} that uses a similar method to the {{OptionalSSLHandler}} of waiting for enough bytes to be received, and then logging a warning about an SSL connection attempt on the port to help the operator spot misconfigurations or security scans.
> There are another couple of related, but minor additional issues.
> The listening message does not currently indicate whether optional secure connections are permitted, it should be explicit about whether secure connections are required, optional and possibly include the internode mode.
> {code}
> INFO [main] 2020-09-15 15:09:33,045 InboundConnectionInitiator.java:128 - Listening on address: (localhost/127.0.0.1:7000), nic: lo0, encryption: disabled
> {code}
> Also {{org.apache.cassandra.security.SSLFactory#buildKeyManagerFactory}} throws {{throw new IOException("failed to build trust manager store for secure connections", e);}} when it should be {{throw new IOException("failed to build key manager store for secure connections", e);}} (probably a copy/paste from {{buildTrustManagerFactory}})
> Proposed changes to resolve this issue and improve behavior:
> * Introduce a DisabledSslHandler that logs a WARN if an SSL connection is attempted when administratively disabled.
> * Add a check that the keystore exists before defaulting optional to false (as a path to a keystore is hard-coded and simply not supplying it does not work)
> * Update the listening message to be explicit about what types of encrypted connections will be accepted.
> * Fix the Keystone IOException to correctly identify the Keystore as the culprit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org