You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/01/29 18:38:34 UTC
[jira] Created: (HBASE-2174) Review how we handle addresses in
HBase
Review how we handle addresses in HBase
---------------------------------------
Key: HBASE-2174
URL: https://issues.apache.org/jira/browse/HBASE-2174
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Fix For: 0.21.0
Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806570#action_12806570 ]
ryan rawson commented on HBASE-2174:
------------------------------------
im wondering if we should do something radical and flexible in this area...
right now if we use IP address, we can make it hard for clients to talk to hbase, since if their knowledge of hbase's IP and the one HBase uses are different, then bam. Think: ec2 internal/external addresses.
Maybe it makes sense to always use symbolic names and resolve them at the client when they need to make RPC calls?
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806455#action_12806455 ]
Jean-Daniel Cryans commented on HBASE-2174:
-------------------------------------------
Another thing people see sometimes is that their region servers will have 2-3 znodes registered in zookeeper, I wasn't able to figure what the problem was right away but the issue exists and it's surely in the scope of this jira.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806580#action_12806580 ]
Joydeep Sen Sarma commented on HBASE-2174:
------------------------------------------
yeah - hostnames are much more flexible. we can review what hdfs does - it already works (for these reasons) in AWS (I was able to get my home pc to talk directly to hdfs nodes inside AWS once).
hadoop also supports configurable hostname per machine (via conf file) (we saw this when we were comparing the hbase and hadoop hostname related code). there's probably some interesting use case for that as well.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837349#action_12837349 ]
stack commented on HBASE-2174:
------------------------------
Patch looks good. I think though that if this patch goes in, to be safe, we should require a restart of hbase upgrading patch versions. So, if we're talking about supporting a rolling restart in 0.20.4, this would go into a 0.20.5.... but its looking like hbase-2248 fix needs to go into 0.20.4 and if so, it dictates a restart.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806611#action_12806611 ]
Kannan Muthukkaruppan commented on HBASE-2174:
----------------------------------------------
Also, in '.META.', the region assignments are stored as IP addresses. Wondering if that could present some "upgrade/compatibility" issues if everything was changed to be hostname based.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837319#action_12837319 ]
Andrew Purtell commented on HBASE-2174:
---------------------------------------
+1 on patch, tests running now, will commit if ok.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2174:
-------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.5
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2174:
-------------------------
Fix Version/s: (was: 0.21.0)
0.20.5
I made a 0.20.5 and moved it here for now. If 0.20.4 requires restart, we'll pull it back in to 0.20.4. Thanks Andrew.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.20.5
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806434#action_12806434 ]
Jean-Daniel Cryans commented on HBASE-2174:
-------------------------------------------
One example of weirdness is when the region server is told which address to use according to the master:
{code}
INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address to use. Was=10.10.21.16:60020, Now=10.10.21.16
{code}
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806561#action_12806561 ]
Kannan Muthukkaruppan commented on HBASE-2174:
----------------------------------------------
JD: The cluster coming down when DNS is flaky (the issue we reported on hbase-dev yesterday) is basically a dup of HBASE-1679. It causes the master to send a "startup" message to an already running region server. This then leads to the region server creating new znodes in zookeeper (of the form /hbase/rs/<startcode>).
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806562#action_12806562 ]
Kannan Muthukkaruppan commented on HBASE-2174:
----------------------------------------------
To fill in some more details, this is what we think happened during DNS flakiness:
A regionServer periodically sends a regionServerReport (RPC call) to the master. A HServerInfo argument is passed as an argument and it identifies the sending region server's identity in IP address format.
The master, in ServerManager class, maintains a serversToServerInfo map which is hostname based. Every time a master receives a regionServerReport it converts the IP address based name to a hostname via the info.getServerName() call. Normally this call returns the hostname, but we suspect that during the DNS flakiness, it returned an IP address based string. And so, this caused ServerManager.java to think that it was hearing from a new server. And this lead to:
HServerInfo storedInfo = serversToServerInfo.get(info.getServerName());
if (storedInfo == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Received report from unknown server -- telling it " + <<============
"to " + CALL_SERVER_STARTUP + ": " + info.getServerName()); <<============
}
and bad things happened down the road (such as the region server registering itself multiple times in Zookeeper, cluster coming down, etc.).
The above error message in our logs (example below) indeed identified the host in IP address syntax, even though normally the getServerName call would return the info in hostname format.
2010-01-28 11:21:34,539 DEBUG org.apache.hadoop.hbase.master.ServerManager: Received report from unknown server -- telling it to MSG_CALL_SERVER_STARTUP: 10.129.68.203,60020,1263605543210
--
Perhaps all we need to do is to change the ServerManager's internal maps to all be IP based? That way we avoid/bypass the master having to look up the hostname on every heartbeat.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837368#action_12837368 ]
Andrew Purtell commented on HBASE-2174:
---------------------------------------
Tests look good. Commit to branch now or wait?
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806608#action_12806608 ]
Karthik Ranganathan commented on HBASE-2174:
--------------------------------------------
@Joydeep/Ryan - good point about EC2. They expose a different IP to the outside world (elastic IP) which is different from the actual machine ip's. We should definitely stick to hostnames.
I guess we should just stop the master from resolving each time and take what the client says its hostname-ip pair is as the truth. Zookeeper currently stores the ip of the root region server - not sure if thats an issue.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Ranganathan updated HBASE-2174:
---------------------------------------
Attachment: HBASE-2174_0.20.3.patch
Add a fix on the HMaster side to stop it from resolving HRegionServer addresses to names using DNS.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2174) Review how we handle addresses in
HBase
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837033#action_12837033 ]
Karthik Ranganathan commented on HBASE-2174:
--------------------------------------------
Hey guys, I have attached a patch for this issue - its a very simple fix on the HMaster side. Do let me know if you have any comments.
> Review how we handle addresses in HBase
> ---------------------------------------
>
> Key: HBASE-2174
> URL: https://issues.apache.org/jira/browse/HBASE-2174
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2174_0.20.3.patch
>
>
> Over the time many parts of the code have evolved in different ways and one issue is that addresses are handled differently in different parts of the code. We need to set a standard and correct any inconsistencies.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.