You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/08/18 00:22:27 UTC
[jira] [Created] (HBASE-4220) Lots of DNS queries from client
Lots of DNS queries from client
-------------------------------
Key: HBASE-4220
URL: https://issues.apache.org/jira/browse/HBASE-4220
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.92.0
In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4220) Lots of DNS queries from client
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088534#comment-13088534 ]
ramkrishna.s.vasudevan commented on HBASE-4220:
-----------------------------------------------
+1
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4220) Lots of DNS queries from client
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087559#comment-13087559 ]
Ted Yu commented on HBASE-4220:
-------------------------------
+1 on patch.
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4220) Lots of DNS queries from client
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4220:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to TRUNK. Thanks for reviews Ted and Ram.
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4220) Lots of DNS queries from client
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4220:
-------------------------
Status: Patch Available (was: Open)
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4220) Lots of DNS queries from client
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-4220:
----------------------------
Assignee: stack
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: stack
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4220) Lots of DNS queries from client
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086654#comment-13086654 ]
Todd Lipcon commented on HBASE-4220:
------------------------------------
I think the constructor of HRegionLocation should create and cache the HServerAddress
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4220) Lots of DNS queries from client
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086643#comment-13086643 ]
Todd Lipcon commented on HBASE-4220:
------------------------------------
I managed to snag the following stack trace:
{noformat}
"pool-1-thread-1" prio=10 tid=0x00002aac380e4800 nid=0x7797 runnable [0x0000000040d4d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00002aabb74d06b8> (a java.lang.NumberFormatException)
at java.lang.Throwable.<init>(Throwable.java:196)
at java.lang.Exception.<init>(Exception.java:41)
at java.lang.RuntimeException.<init>(RuntimeException.java:43)
at java.lang.IllegalArgumentException.<init>(IllegalArgumentException.java:36)
at java.lang.NumberFormatException.<init>(NumberFormatException.java:38)
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
at java.lang.Integer.parseInt(Integer.java:499)
at sun.net.util.IPAddressUtil.textToNumericFormatV4(IPAddressUtil.java:94)
at java.net.InetAddress.getAllByName(InetAddress.java:1051)
at java.net.InetAddress.getAllByName(InetAddress.java:1020)
at java.net.InetAddress.getByName(InetAddress.java:970)
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:124)
at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:75)
at org.apache.hadoop.hbase.HRegionLocation.getServerAddress(HRegionLocation.java:101)
at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:71)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1238)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:612)
{noformat}
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4220) Lots of DNS queries from client
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4220:
-------------------------
Attachment: 4220.txt
Rather than cache a HSA, I just looked at who uses getServerAddress and as it happens, no one needs to be using getServerAddress. Only in one case does it make a difference in that the HSA.toString that was being used now is hostname+port only whereas it used be hostname/ip+port; i.e. InetAddress.toString()+port. Shouldn't matter.
> Lots of DNS queries from client
> -------------------------------
>
> Key: HBASE-4220
> URL: https://issues.apache.org/jira/browse/HBASE-4220
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4220.txt
>
>
> In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira