You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/26 19:25:23 UTC
[GitHub] [pulsar] evfurman opened a new issue #10388: Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record
evfurman opened a new issue #10388:
URL: https://github.com/apache/pulsar/issues/10388
**Describe the bug**
Bookies unable to connect to ZK over TLS on Pulsar version `2.7.1`.
**To Reproduce**
Steps to reproduce the behavior:
1. Deploy Pulsar cluster version `2.7.1`
2. Configure certificate-based TLS/SSL cluster-wide.
3. Bookies fail to initialize metadata drive and `pulsar-admin` commands return a 500 errror.
**Expected behavior**
Bookies should be able to connect to Zookeeper nodes over TLS/SSL without issue.
**Additional context**
```
# BOOKIE LOGS
15:11:55.930 [main] ERROR org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Failed to create zookeeper client to zk-1.pluster-271.qa.example.com:2281,zk-2.pluster-271.qa.example.com:2281,zk-3.pluster-271.qa.example.com:2281 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260)
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197)
at org.apache.bookkeeper.meta.zk.ZKMetadataBookieDriver.initialize(ZKMetadataBookieDriver.java:60)
at org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1093)
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:711)
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
at org.apache.bookkeeper.server.Main.main(Main.java:208)
15:11:55.957 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to initialize metadata bookie driver
at org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1103)
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:711)
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
at org.apache.bookkeeper.server.Main.main(Main.java:208)
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to zk-1.pluster-271.qa.example.com:2281,zk-2.pluster-271.qa.example.com:2281,zk-3.pluster-271.qa.example.com:2281
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:217)
at org.apache.bookkeeper.meta.zk.ZKMetadataBookieDriver.initialize(ZKMetadataBookieDriver.java:60)
at org.apache.bookkeeper.bookie.Bookie.instantiateMetadataDriver(Bookie.java:1093)
... 7 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159)
# ZOOKEEPER LOGS
18:55:53.973 [CommitProcWorkThread-2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x1000017dd3e0007 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
18:55:53.973 [CommitProcWorkThread-2] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - sessionid:0x1000017dd3e0007 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG org.apache.zookeeper.server.NettyServerCnxnFactory - SSL handler added for channel: [id: 0x66fb966c, L:/10.3.21.69:2281 - R:/10.3.22.228:35764]
18:55:54.620 [epollEventLoopGroup-7-2] ERROR org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake with session 0x0
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG org.apache.zookeeper.server.NettyServerCnxn - close called for sessionid:0x0
18:55:54.620 [epollEventLoopGroup-7-2] DEBUG org.apache.zookeeper.server.NettyServerCnxn - cnxns size:4
18:55:54.620 [epollEventLoopGroup-7-2] WARN org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
... 15 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] evfurman commented on issue #10388: Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record
Posted by GitBox <gi...@apache.org>.
evfurman commented on issue #10388:
URL: https://github.com/apache/pulsar/issues/10388#issuecomment-827123614
It looks like the pulsar command script changed which set of options get passed to the "bookie" command specifically. It used to collect from `PULSAR_EXTRA_OPTS` but seems to skip those now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] evfurman closed issue #10388: Exception caught io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record
Posted by GitBox <gi...@apache.org>.
evfurman closed issue #10388:
URL: https://github.com/apache/pulsar/issues/10388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org