You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/09/08 16:23:22 UTC
[GitHub] [pulsar] cdbartholomew opened a new pull request #5151: Broker
fails to start with function worker enabled and broker client using TLS
cdbartholomew opened a new pull request #5151: Broker fails to start with function worker enabled and broker client using TLS
URL: https://github.com/apache/pulsar/pull/5151
### Motivation
I have a broker that is configured to run the function worker and for the broker client to use TLS. Here are the settings:
functionsWorkerEnabled=true
brokerClientTlsEnabled=true
After upgrading to 2.4.1, my broker would fail to start. This was working in 2.3.1. The function worker client would fail to connect. The error indicates that it is trying to use TLS on the plaintext port:
12:39:20.691 [main] INFO org.apache.pulsar.functions.worker.WorkerService - Created Pulsar client
12:39:21.057 [pulsar-io-24-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.34.192:60886
12:39:21.045 [pulsar-client-io-47-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x688be07c, L:/192.168.34.192:60886 - R:192.168.34.192/192.168.34.192:6650]] Connected to server
12:39:21.080 [pulsar-io-24-1] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.34.192:60886] Got exception TooLongFrameException : Adjusted frame length exceeds 5253120: 369295620 - discarded
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 369295620 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:522) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:500) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:387) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:430) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:343) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
### Modifications
Looking at the startup code when running the function worker with the broker, it is checking for TLS enabled in the function_worker.yml file to determine whether or not to use the TLS port, but when setting TLS enabled on the function worker, it is checking the broker.conf.
Since the function worker is running with the broker, it makes sense to look to the broker.conf as the single source of truth about whether or not to use TLS. I changed the code to check the broker client is configured to use TLS. If it is, then use TLS for the function worker, otherwise use plain text.
The same code exists in the startup path for the normal broker and the standalone version, so I made the change in both places.
### Verifying this change
I ran the broker unit tests on this change and they all passed.
I also built a new docker image with this change and started using it in my cluster. The broker now starts successfully.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): no
- The public API: no
- The schema: no
- The default values of configurations: no
- The wire protocol: no
- The rest endpoints: no
- The admin cli options: no
- Anything that affects deployment: no
### Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services