You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2017/08/19 00:31:00 UTC
[jira] [Commented] (KUDU-1894) ITClient very flaky since
introduction of TLS
[ https://issues.apache.org/jira/browse/KUDU-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133834#comment-16133834 ]
Todd Lipcon commented on KUDU-1894:
-----------------------------------
Managed to repro a deadlock using dist-test loop (1/700 deadlocked) on commit a15f1863e42fb308fcb13dba651f72d1cbe9e1c4:
{code}
Found one Java-level deadlock:
=============================
"Thread-9":
waiting to lock monitor 0x00007ff20401cbb8 (object 0x0000000775e109d8, a java.lang.Object),
which is held by "New I/O worker #24"
"New I/O worker #24":
waiting for ownable synchronizer 0x0000000775dff920, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "Thread-9"
Java stack information for the threads listed above:
===================================================
"Thread-9":
at org.jboss.netty.handler.ssl.SslHandler.wrap(SslHandler.java:916)
- waiting to lock <0x0000000775e109d8> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:557)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at org.apache.kudu.client.Connection.sendCallToWire(Connection.java:576)
at org.apache.kudu.client.Connection.enqueueMessage(Connection.java:481)
at org.apache.kudu.client.RpcProxy.sendRpc(RpcProxy.java:127)
at org.apache.kudu.client.AsyncKuduClient.sendRpcToTablet(AsyncKuduClient.java:871)
at org.apache.kudu.client.AsyncKuduScanner.nextRows(AsyncKuduScanner.java:448)
at org.apache.kudu.client.KuduScanner.nextRows(KuduScanner.java:58)
at org.apache.kudu.client.ITClient$ScannerThread.randomGet(ITClient.java:354)
at org.apache.kudu.client.ITClient$ScannerThread.run(ITClient.java:321)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #24":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000775dff920> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.apache.kudu.client.Connection.cleanup(Connection.java:592)
at org.apache.kudu.client.Connection.channelDisconnected(Connection.java:245)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.apache.kudu.client.Connection.handleUpstream(Connection.java:236)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:580)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
at org.jboss.netty.channel.Channels.close(Channels.java:828)
at org.jboss.netty.handler.ssl.SslHandler.handleRenegotiation(SslHandler.java:1374)
- locked <0x0000000775e109d8> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248)
- locked <0x0000000775e109d8> (a java.lang.Object)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
{code}
> ITClient very flaky since introduction of TLS
> ---------------------------------------------
>
> Key: KUDU-1894
> URL: https://issues.apache.org/jira/browse/KUDU-1894
> Project: Kudu
> Issue Type: Bug
> Components: java
> Affects Versions: 1.3.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: n/a
>
>
> Since introducing TLS support in the Java client, ITClient seems to hang/timeout quite often. It always spews some error about TLS renegotiation, but it seems like that error description is incorrect -- really it seems more likely that something is amiss with the way this test tries to shut down client connections by reaching into the Netty channel.
> Either way we should investigate for 1.3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)