[jira] [Resolved] (HBASE-23731) TestFromClientSide flakey after junit upgrade

["Bharath Vissapragada (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-23731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Vissapragada resolved HBASE-23731.
------------------------------------------
    Fix Version/s: HBASE-18095
       Resolution: Fixed

Thanks [~stack] and [~janh] for the review.

> TestFromClientSide flakey after junit upgrade
> ---------------------------------------------
>
>                 Key: HBASE-23731
>                 URL: https://issues.apache.org/jira/browse/HBASE-23731
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: HBASE-18095
>            Reporter: Bharath Vissapragada
>            Assignee: Bharath Vissapragada
>            Priority: Major
>              Labels: flaky-test
>             Fix For: HBASE-18095
>
>
> The cluster shutdown and restart between parameterized runs causes a thread leak with stacks like follows.
> {noformat}
> 2020-01-22 09:52:10,900 WARN  [LeaseRenewer:hostname@localhost:62252] impl.LeaseRenewer(438): Failed to renew lease for [DFSClient_NONMAPREDUCE_-1123200667_19] for 73 seconds.  Will retry shortly ...
> java.net.ConnectException: Call From xxxxx/xxxx to localhost:62252 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> 	at sun.reflect.GeneratedConstructorAccessor64.newInstance(Unknown Source)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
> 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1435)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1345)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 	at com.sun.proxy.$Proxy29.renewLease(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:613)
> 	at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> 	at com.sun.proxy.$Proxy30.renewLease(Unknown Source)
> 	at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
> 	at com.sun.proxy.$Proxy33.renewLease(Unknown Source)
> 	at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
> 	at com.sun.proxy.$Proxy33.renewLease(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:560)
> 	at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.renew(LeaseRenewer.java:395)
> 	at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:415)
> 	at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
> 	at org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307)
> 	at java.lang.Thread.run(Thread.java:748)
> {noformat}
> While the issue existed even before, it looks like junit 4.12 somehow was masking it. Upgrade to 4.13 made it worse.  The host:port refers to already shutdown hdfs cluster, so ideally this thread shouldn't be there in the first place. A culprit could be leaking filesystem / dfsclient objects.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)