You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nick Dimiduk (Jira)" <ji...@apache.org> on 2019/08/21 18:37:00 UTC
[jira] [Commented] (HBASE-22169) Open region failed cause memory
leak
[ https://issues.apache.org/jira/browse/HBASE-22169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912589#comment-16912589 ]
Nick Dimiduk commented on HBASE-22169:
--------------------------------------
[~luffy123], [~stack] {{TestHRegion#testOpenRegionFailedMemoryLeak}} fails consistently (2/3 times) for me on 2.1.6RC0 with the following stacktrace. I think you have a race between the {{Time-Limited test}} timeout and the amount of time the test waits for metrics to propagate. It'll timeout if the amount of time spent by other test methods before this test is run eat up the overall class timeout quota. For this reason, I suggest extracting this method out into it's own dedicated class.
FYI [~Apache9] .
{noformat}
2019-08-21 11:15:34,239 ERROR [Time-limited test] coprocessor.CoprocessorHost(398): The coprocessor test threw java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException; Host Details : local host is: "myson.local/192.168.111.14"; destination host is: "test":8020;
java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException; Host Details : local host is: "myson.local/192.168.111.14"; destination host is: "test":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy38.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy39.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2117)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1451)
at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:275)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:827)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:716)
at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:7008)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7218)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7191)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7149)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7100)
at org.apache.hadoop.hbase.regionserver.TestHRegion.testOpenRegionFailedMemoryLeak(TestHRegion.java:6623)
{noformat}
> Open region failed cause memory leak
> ------------------------------------
>
> Key: HBASE-22169
> URL: https://issues.apache.org/jira/browse/HBASE-22169
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.0.5, 2.1.4
> Reporter: Bing Xiao
> Assignee: Bing Xiao
> Priority: Critical
> Fix For: 3.0.0, 2.2.1, 2.1.6
>
> Attachments: 0001-HBASE-22169-Open-region-failed-cause-memory-leak.patch, HBASE-22169-branch-2-v1.patch, HBASE-22169-branch-2.1-v1.patch, HBASE-22169-branch-2.2-v1.patch, HBASE-22169-master-v1.patch, HBASE-22169-master-v2.patch, HBASE-22169-master-v3.patch
>
>
> In some cases (for example, coprocessor path is wrong) region open failed, MetricsRegionWrapperImpl is already init and not close, cause memory leak;
> {code:java}
> 2019-02-21 15:41:32,929 ERROR [RS_OPEN_REGION-hb-2zedsc3fxjn12dl6u-005:16020-7] regionserver.RegionCoprocessorHost(362): Failed to load coprocessor org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService
> java.lang.IllegalArgumentException: java.net.UnknownHostException: emr-cluster
> at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
> at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
> at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:165)
> at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:194)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:352)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:240)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:749)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:657)
> at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6727)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7037)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7009)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6965)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6916)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129){code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)