You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Karthik Ranganathan (Created) (JIRA)" <ji...@apache.org> on 2011/10/31 15:51:33 UTC
[jira] [Created] (HBASE-4707) Avoid unnecessary DNS lookups for
locality computation in HMaster
Avoid unnecessary DNS lookups for locality computation in HMaster
-----------------------------------------------------------------
Key: HBASE-4707
URL: https://issues.apache.org/jira/browse/HBASE-4707
Project: HBase
Issue Type: Bug
Components: master
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan
We recently hit an issue where upon each RS heartbeat we were looking up and resolving DNS name + address of the RS in the master, but needed it only for locality based assignment on startup.
Some flakiness in the DNS subsystem cause one of the threads to get stuck in the lookup and the synchronized call at:
ServerManager.java:528
processMsgs() {
...
synchronized (this.master.getRegionManager()) {
// does dns lookup
}
}
The offending stack trace was:
"IPC Server handler 232 on 60000" daemon prio=10 tid=0x00007fcb64164000 nid=0x7d16 runnable [0x0000000052e7f000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
at java.net.InetAddress.getHostFromNameService(InetAddress.java:550)
at java.net.InetAddress.getHostName(InetAddress.java:476)
at java.net.InetAddress.getHostName(InetAddress.java:448)
at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
at org.apache.hadoop.hbase.HServerAddress.getHostname(HServerAddress.java:117)
at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:469)
at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:263)
at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:500)
- locked <0x00007fcb985b2030> (a org.apache.hadoop.hbase.master.RegionManager)
at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:841)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:585)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:933)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4707) Avoid unnecessary DNS lookups for
locality computation in HMaster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443416#comment-13443416 ]
stack commented on HBASE-4707:
------------------------------
Agree to closing this one. We don't have HServerAddress inline w/ regionserver report -- we have ServerName which doesn't do DNS lookups -- and the stack trace is no longer possible, its stale, from the RegionManager on down (there is no RegionManager class in hbase)
> Avoid unnecessary DNS lookups for locality computation in HMaster
> -----------------------------------------------------------------
>
> Key: HBASE-4707
> URL: https://issues.apache.org/jira/browse/HBASE-4707
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> We recently hit an issue where upon each RS heartbeat we were looking up and resolving DNS name + address of the RS in the master, but needed it only for locality based assignment on startup.
> Some flakiness in the DNS subsystem cause one of the threads to get stuck in the lookup and the synchronized call at:
> ServerManager.java:528
> processMsgs() {
> ...
> synchronized (this.master.getRegionManager()) {
> // does dns lookup
> }
> }
> The offending stack trace was:
> "IPC Server handler 232 on 60000" daemon prio=10 tid=0x00007fcb64164000 nid=0x7d16 runnable [0x0000000052e7f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
> at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
> at java.net.InetAddress.getHostFromNameService(InetAddress.java:550)
> at java.net.InetAddress.getHostName(InetAddress.java:476)
> at java.net.InetAddress.getHostName(InetAddress.java:448)
> at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
> at org.apache.hadoop.hbase.HServerAddress.getHostname(HServerAddress.java:117)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:469)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:263)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:500)
> - locked <0x00007fcb985b2030> (a org.apache.hadoop.hbase.master.RegionManager)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:841)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:585)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:933)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4707) Avoid unnecessary DNS lookups for
locality computation in HMaster
Posted by "Jesse Yates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jesse Yates resolved HBASE-4707.
--------------------------------
Resolution: Fixed
Closing as fixed in trunk. Reopen if I'm mistaken.
> Avoid unnecessary DNS lookups for locality computation in HMaster
> -----------------------------------------------------------------
>
> Key: HBASE-4707
> URL: https://issues.apache.org/jira/browse/HBASE-4707
> Project: HBase
> Issue Type: Bug
> Components: master
> Reporter: Karthik Ranganathan
> Assignee: Karthik Ranganathan
>
> We recently hit an issue where upon each RS heartbeat we were looking up and resolving DNS name + address of the RS in the master, but needed it only for locality based assignment on startup.
> Some flakiness in the DNS subsystem cause one of the threads to get stuck in the lookup and the synchronized call at:
> ServerManager.java:528
> processMsgs() {
> ...
> synchronized (this.master.getRegionManager()) {
> // does dns lookup
> }
> }
> The offending stack trace was:
> "IPC Server handler 232 on 60000" daemon prio=10 tid=0x00007fcb64164000 nid=0x7d16 runnable [0x0000000052e7f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
> at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
> at java.net.InetAddress.getHostFromNameService(InetAddress.java:550)
> at java.net.InetAddress.getHostName(InetAddress.java:476)
> at java.net.InetAddress.getHostName(InetAddress.java:448)
> at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
> at org.apache.hadoop.hbase.HServerAddress.getHostname(HServerAddress.java:117)
> at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:469)
> at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:263)
> at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:500)
> - locked <0x00007fcb985b2030> (a org.apache.hadoop.hbase.master.RegionManager)
> at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
> at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
> at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:841)
> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:585)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:933)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira