You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/05/10 17:10:32 UTC
[GitHub] [accumulo] sjyang18 opened a new issue #2092: When TLS is enabled in the zookeeper side, accumulo monitor should use TLS connection to get zookeeper status.
sjyang18 opened a new issue #2092:
URL: https://github.com/apache/accumulo/issues/2092
**Describe the bug**
When TLS is enabled & configured in the zookeepers, accumulo monitor still continuously use normal Transport connections to zookeepers to get zookeeper status. This generates unsuccessful handshake errors and NotSslRecordException in zookeepers continuously. Looking at code, TTimeoutTransport class seems to be used, which fails to do handshaking with TLS-enabled zookeepers.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: [e.g. 1.10.0] 2.1.0
- OS: [e.g. CentOS 7.5]
- Others:
**To Reproduce**
Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):
1. Turn TLS on zookeepers. And, configure Accumulo to use TLS as documented.
2. Add additional zookeeper client libraries and CLIENT_JVMFLAGS to accumulo-env.sh
```
add_jar_prefix_to_classpath "${ZOOKEEPER_HOME}/lib/netty-"
CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \
-Dzookeeper.client.secure=true \
-Dzookeeper.ssl.keyStore.location=/opt/muchos/install/ssl/host-keystore.jks \
-Dzookeeper.ssl.keyStore.password=hadoop \
-Dzookeeper.ssl.trustStore.location=/opt/muchos/install/ssl/truststore.jks \
-Dzookeeper.ssl.trustStore.password=hadoop"
JAVA_OPTS=("${JAVA_OPTS[@]}" $CLIENT_JVMFLAGS)
```
3. accumulo-cluster start
4. Tail the zookeeper log and observer 'Unsuccessful handshake with session and io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Accumulo monitor thread who calls ZookeeperStatus.
```
"ZooKeeperStatus" #63 daemon prio=5 os_prio=0 tid=0x000055f643762000 nid=0x4351 waiting on condition [0x00007f3cadb49000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at org.apache.accumulo.fate.util.UtilWaitThread.sleepUninterruptibly(UtilWaitThread.java:55)
at org.apache.accumulo.monitor.ZooKeeperStatus.run(ZooKeeperStatus.java:151)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
```
Sample errors and exceptions in zookeeper logs.
```
2021-05-10 16:30:45,664 [myid:1] - ERROR [nioEventLoopGroup-4-49:NettyServerCnxnFactory$CertificateVerifier@363] - Unsuccessful handshake with session 0x0
2021-05-10 16:30:45,665 [myid:1] - WARN [nioEventLoopGroup-4-49:NettyServerCnxnFactory$CnxnChannelHandler@220] - Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737461740a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737461740a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
... 17 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii closed issue #2092: When TLS is enabled in the zookeeper side, accumulo monitor should use TLS connection to get zookeeper status.
Posted by GitBox <gi...@apache.org>.
ctubbsii closed issue #2092:
URL: https://github.com/apache/accumulo/issues/2092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2092: When TLS is enabled in the zookeeper side, accumulo monitor should use TLS connection to get zookeeper status.
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2092:
URL: https://github.com/apache/accumulo/issues/2092#issuecomment-837289892
@EdColeman It looks like the view was removed from the monitor, but the background thread that does the 4lw operation still tries to run. See the ZooKeeperStatus class and its background thread in the Monitor class. These should probably be removed to complete the work began in #2020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2092: When TLS is enabled in the zookeeper side, accumulo monitor should use TLS connection to get zookeeper status.
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2092:
URL: https://github.com/apache/accumulo/issues/2092#issuecomment-837002418
The zookeeper status has been removed from the monitor, starting with 2.1 (when that is released)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2092: When TLS is enabled in the zookeeper side, accumulo monitor should use TLS connection to get zookeeper status.
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2092:
URL: https://github.com/apache/accumulo/issues/2092#issuecomment-839185577
This is OBE by #2020 and #2095, which removed the ZK status thread, REST endpoint, and web view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org