You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Alex Newman (JIRA)" <ji...@apache.org> on 2009/04/29 20:39:30 UTC
[jira] Created: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
-------------------------------------------------------------------------------------
Key: HBASE-1357
URL: https://issues.apache.org/jira/browse/HBASE-1357
Project: Hadoop HBase
Issue Type: Bug
Components: master, regionserver
Affects Versions: 0.20.0, 0.20.1, 0.21.0
Environment: All
Reporter: Alex Newman
Fix For: 0.20.0, 0.20.1, 0.21.0
(2:11:20 PM) posix4e: so i want to run a back master on each node
(2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
(2:14:59 PM) posix4e: each master only gets the local regionserver connecting
(2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
(2:15:32 PM) nitay: the RS don't use hbase.master* anymore
(2:15:36 PM) nitay: ohhh i think i know th eproblem
(2:15:44 PM) nitay: so the RS use ZK to get the master address
(2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
(2:15:58 PM) nitay: b/c they write whatever was in their conf
(2:16:20 PM) posix4e: yea
(2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
(2:16:55 PM) posix4e: yea
(2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
(2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
(2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
(2:19:47 PM) posix4e: is their a way to do that?
(2:20:16 PM) nitay: i dont know, good question
(2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
(2:20:27 PM) nitay: did u verify the master address?
(2:20:48 PM) posix4e: one sec
(2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
(2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
(2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
(2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
(2:22:40 PM) nitay: yep that's the magic
(2:22:45 PM) nitay: k thx for verifying
(2:22:54 PM) nitay: u want to open up a JIRA?
(2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
(2:22:58 PM) posix4e: yea
(2:23:09 PM) posix4e: can i quote this conversation?
(2:23:18 PM) nitay: yes please do
(2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
(2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
(2:24:02 PM) nitay: i know, its a royal PITA
(2:24:59 PM) posix4e: yea
(2:25:03 PM) posix4e: no problem
(2:25:20 PM) nitay: but that should work till we find a better solution
(2:25:21 PM) posix4e: I am trying to think how a patch would work
(2:25:25 PM) posix4e: have a masters file?
(2:25:44 PM) nitay: yeah if u have any ideas please offer them
(2:25:46 PM) nitay: hmm interesting idea
(2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
(2:26:26 PM) posix4e: yea
(2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
(2:28:30 PM) nitay: but i like your idea
(2:28:37 PM) nitay: post it on the JIRA
(2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
(2:32:56 PM) posix4e: yep
I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1357:
--------------------------------------
Attachment: hbase-1357-v3.patch
Patch adds some clarification in the doc, adds hbase.master.dns.nameserver and hbase.master.dns.interface to specify which interface the master should use to communicate with which DNS (just like in hadoop).
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch, hbase-1357-v3.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711845#action_12711845 ]
Alex Newman commented on HBASE-1357:
------------------------------------
We use the same script which restarts all of the zookeeper , thrift ,
master , region and rsync nodes to rewrite the hbase master address.
It's hacky but it works. The problem is the zookeeper node. I think
the way you do that in zookeeper is with a sequence and a watch (I
think). I would code this up but I am in a car on my handy. If no one
has it fixed by Tuesday I will talk to Nitay about it.
--
Sent from my mobile device
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1357:
----------------------------
Assignee: Nitay Joffe
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-1357:
-----------------------------------------
Assignee: Jean-Daniel Cryans (was: Nitay Joffe)
Nitay agreed to let me work on the problem since it's related to HBASE-1445.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1357:
-------------------------
Fix Version/s: (was: 0.21.0)
0.20.0
Boys chatting on #hbase today figure we should do this -- just remove master address and do lookup of host and write that to ZK. Bringing it back into 0.20.0. Nitay says assign it to him.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712111#action_12712111 ]
Jean-Daniel Cryans commented on HBASE-1357:
-------------------------------------------
Ok let's do it like that. One thing that we should do tho is to have a more detailed "How to run distributed mode" documentation.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713196#action_12713196 ]
Jean-Daniel Cryans commented on HBASE-1357:
-------------------------------------------
Chatting with Nitay, we figured it would be best to just have a "cluster mode" configuration that tells HBase whether it's local or distributed instead of the master address since it won't be needed anymore.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715582#action_12715582 ]
stack commented on HBASE-1357:
------------------------------
+1. Commit it.
I tested it. Makes sense. Doc. will need rework but lets get all the other bits in there first before we do that.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch, hbase-1357-v3.patch, hbase-1357-v4.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714779#action_12714779 ]
stack commented on HBASE-1357:
------------------------------
The choice of configuration name and setting is important since it'll be one of the first things presented a user.
hbase.cluster.mode with 'local' | 'distributed' values seems a little awkward to me (and error-prone).
What about hbase.cluster.distributed with a yes/no or true/false answer (I think Configuration will do right thing whatever you provide).
If 'false', then hbase cluster is not distributed across a cluster of machines -- its standalone or pseudo-distributed.
The description in hbase-default needs to be a little clearer. Also, I'm not too clear on what happens. If distributed, it means user needs to edit zoo.cfg to point at quorum? Where is the switch for whether or not hbase starts up the quorum?
Otherwise, patch looks great. I like --master option to PE. I'd also remove the commented out code in HConstants.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1357:
-------------------------------
Description:
(2:11:20 PM) posix4e: so i want to run a back master on each node
(2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
(2:14:59 PM) posix4e: each master only gets the local regionserver connecting
(2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
(2:15:32 PM) nitay: the RS don't use hbase.master* anymore
(2:15:36 PM) nitay: ohhh i think i know th eproblem
(2:15:44 PM) nitay: so the RS use ZK to get the master address
(2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
(2:15:58 PM) nitay: b/c they write whatever was in their conf
(2:16:20 PM) posix4e: yea
(2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
(2:16:55 PM) posix4e: yea
(2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
(2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
(2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
(2:19:47 PM) posix4e: is their a way to do that?
(2:20:16 PM) nitay: i dont know, good question
(2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
(2:20:27 PM) nitay: did u verify the master address?
(2:20:48 PM) posix4e: one sec
(2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
(2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
(2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
(2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
(2:22:40 PM) nitay: yep that's the magic
(2:22:45 PM) nitay: k thx for verifying
(2:22:54 PM) nitay: u want to open up a JIRA?
(2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
(2:22:58 PM) posix4e: yea
(2:23:09 PM) posix4e: can i quote this conversation?
(2:23:18 PM) nitay: yes please do
(2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
(2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
(2:24:02 PM) nitay: i know, its a royal PITA
(2:24:59 PM) posix4e: yea
(2:25:03 PM) posix4e: no problem
(2:25:20 PM) nitay: but that should work till we find a better solution
(2:25:21 PM) posix4e: I am trying to think how a patch would work
(2:25:25 PM) posix4e: have a masters file?
(2:25:44 PM) nitay: yeah if u have any ideas please offer them
(2:25:46 PM) nitay: hmm interesting idea
(2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
(2:26:26 PM) posix4e: yea
(2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
(2:28:30 PM) nitay: but i like your idea
(2:28:37 PM) nitay: post it on the JIRA
(2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
(2:32:56 PM) posix4e: yep
I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
was:
(2:11:20 PM) posix4e: so i want to run a back master on each node
(2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
(2:14:59 PM) posix4e: each master only gets the local regionserver connecting
(2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
(2:15:32 PM) nitay: the RS don't use hbase.master* anymore
(2:15:36 PM) nitay: ohhh i think i know th eproblem
(2:15:44 PM) nitay: so the RS use ZK to get the master address
(2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
(2:15:58 PM) nitay: b/c they write whatever was in their conf
(2:16:20 PM) posix4e: yea
(2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
(2:16:55 PM) posix4e: yea
(2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
(2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
(2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
(2:19:47 PM) posix4e: is their a way to do that?
(2:20:16 PM) nitay: i dont know, good question
(2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
(2:20:27 PM) nitay: did u verify the master address?
(2:20:48 PM) posix4e: one sec
(2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
(2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
(2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
(2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
(2:22:40 PM) nitay: yep that's the magic
(2:22:45 PM) nitay: k thx for verifying
(2:22:54 PM) nitay: u want to open up a JIRA?
(2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
(2:22:58 PM) posix4e: yea
(2:23:09 PM) posix4e: can i quote this conversation?
(2:23:18 PM) nitay: yes please do
(2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
(2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
(2:24:02 PM) nitay: i know, its a royal PITA
(2:24:59 PM) posix4e: yea
(2:25:03 PM) posix4e: no problem
(2:25:20 PM) nitay: but that should work till we find a better solution
(2:25:21 PM) posix4e: I am trying to think how a patch would work
(2:25:25 PM) posix4e: have a masters file?
(2:25:44 PM) nitay: yeah if u have any ideas please offer them
(2:25:46 PM) nitay: hmm interesting idea
(2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
(2:26:26 PM) posix4e: yea
(2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
(2:28:30 PM) nitay: but i like your idea
(2:28:37 PM) nitay: post it on the JIRA
(2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
(2:32:56 PM) posix4e: yep
I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
Fix Version/s: (was: 0.20.1)
(was: 0.20.0)
Moving out of 0.20 for now. In meeting we discussed:
- Getting rid of hbase.master.hostname property completely. Detect master address using an inet address detection scheme similar to what Hadoop does.
- The above breaks the default ZooKeeper case of a single server running on the master node. RegionServers currently find this ZooKeeper using the master address property. This default case should be replaced by having the zoo.cfg turn into a template file which can be created and rsync'ed to RegionServer nodes by e.g. start-hbase.sh script.
- Add a masters file, similar to regionservers, with pool of servers for master election.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Fix For: 0.21.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711857#action_12711857 ]
stack commented on HBASE-1357:
------------------------------
Chatting w/ Nitay, he recalled that reason this issue was punted to 0.21 was because that in distributed hbase -- not standalone nor pseudo-distributed -- then clients and regionservers need to know where the zookeeper quorum is. This means edit of zoo.cfg WHEN YOU WANT TO RUN IN DISTRIBUTED MODE. Chatting w/ Nitay, I thought we could continue hiding ZK from noobs by doing something like adding a new property in hbase-site.xml named zookeeper.quorum and in it we'd list all quorum members and then in background we'd write the zoo.cfg but Nitay just raised his eyebrow when i suggested this.
So, I'm with him now. Lets not beat around the bush. When doing distributed mode, then you need to edit the ZK config. Will reinforce that ZK is cluster mediator. I think its fine. The two basic modes out of the box will just work w/o zoo.cfg edits. What you think J-D?
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1357:
--------------------------------------
Attachment: hbase-1357-v1.patch
First rough cut. There is no documentation tho the Getting Started will have to change a bit.
I removed hbase.master.hostname and added a cluster mode. ZK complains and exits if mode is distributed but it's still managed by HBase like this:
{quote}
2009-05-28 11:34:58,288 FATAL org.apache.hadoop.hbase.zookeeper.HQuorumPeer: Zookeeper should be managed only in a local cluster mode. Please edit conf/zoo.cfg and remove hbase.cluster.mode to set your ZK quorum addresses
{quote}
It passes test tho, and I tried PE in standalone, pseudo and fully distributed. All works.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1357:
--------------------------------------
Attachment: hbase-1357-v2.patch
In this patch I changed hbase.cluster.mode to hbase.cluster.distributed with options false|true. I also added new documentation in overview.html to explain the new steps for the fully-distributed setup.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715276#action_12715276 ]
stack commented on HBASE-1357:
------------------------------
I still get this:
{code}
2009-06-01 21:29:19,071 ERROR org.apache.hadoop.hbase.master.HMaster: Can not start master
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1090)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1127)
Caused by: java.io.IOException: Could not read quorum servers from zoo.cfg
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.<init>(ZooKeeperWrapper.java:90)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.<init>(ZooKeeperWrapper.java:78)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:245)
... 6 more
{code}
Its because I still have this in my zoo.cfg:
{code}
server.0=${hbase.cluster.distributed}:2888:3888
{code}
Its odd having a variable whose setttings are true/false in this position.
It should be 'hbase.hostname', something we calculate for you when in local/pseudo mode putting in place hostname unless its overridden.
Exception needs to be better.
We can't put localhost here? If hbase.cluster.distributed is true and this zoo.cfg is localhost then throw exception? Would that work?
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch, hbase-1357-v3.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1357.
---------------------------------------
Resolution: Fixed
Release Note:
hbase.master and hbase.master.hostname are now obsolete. hbase.cluster.distributed must be set at "true" to have a fully-distributed setup along with at least one configured ZK server which is not pointing at localhost.
Also, zoo.cfg must be in the classpath of every client.
Hadoop Flags: [Incompatible change, Reviewed]
Committed to trunk.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch, hbase-1357-v3.patch, hbase-1357-v4.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1357) If one sets the hbase.master to
0.0.0.0 non local regionservers can't find the master
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1357:
--------------------------------------
Attachment: hbase-1357-v4.patch
This latest version of the patch clears up hbase.cluster.distributed from zoo.cfg like Stack described. This will be thrown when the hbase.cluster.distributed is true and the value in zoo.cfg is localhost:
{quote}
localhost: starting zookeeper, logging to /home/jdcryans/svn/hbase/trunk/bin/../logs/hbase-jdcryans-zookeeper-jdcryans.mtl.out
localhost: java.io.IOException: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See "Getting Started" for more information.
localhost: at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.parseConfig(HQuorumPeer.java:141)
localhost: at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.parseZooKeeperConfig(HQuorumPeer.java:82)
localhost: at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.main(HQuorumPeer.java:58)
{quote}
I tested master failover on 3 nodes doing hbase-daemons.sh start master then regionserver (which is kinda fun) and killed the first, then the second master. What I first saw is all regions getting reassigned (which was supposed to be fix) but this was because of the alls well messages:
{quote}
2009-06-02 11:02:19,941 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
2009-06-02 11:02:19,942 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 62000: starting
2009-06-02 11:02:19,956 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 192.168.1.88:62020, regionname: -ROOT-,,0, startKey: <>} complete
2009-06-02 11:02:20,144 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 54 row(s) of meta region {server: 192.168.1.87:62020, regionname: .META.,,1, startKey: <>} complete
2009-06-02 11:02:20,939 DEBUG org.apache.hadoop.hbase.master.ServerManager: Process all wells: address: 192.168.1.88:62020, startcode: 1243954293421, load: (requests=4, regions=19, usedHeap=27, maxHeap=963) openingCount: 0, nobalancingCount: 4
2009-06-02 11:02:20,945 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Wrote out of safe mode
2009-06-02 11:02:20,945 INFO org.apache.hadoop.hbase.master.RegionManager: exiting safe mode
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server is overloaded. Server load: 19 avg: 6.333333333333333, slop: 0.1
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 12 regions. mostLoadedRegions has 10 regions in it.
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0003047546,1242765850701
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0011562998,1241636821854
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0006799914,1242765888384
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0006349363,1242765888384
2009-06-02 11:02:20,954 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0001399792,1242840206758
2009-06-02 11:02:20,955 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0000072704,1242764140942
2009-06-02 11:02:20,955 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0008901015,1242765794486
2009-06-02 11:02:20,955 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0002902745,1242765697441
2009-06-02 11:02:20,955 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0000283600,1242764792449
2009-06-02 11:02:20,955 DEBUG org.apache.hadoop.hbase.master.RegionManager: Going to close region TestTable,0000612479,1242394901854
2009-06-02 11:02:20,955 INFO org.apache.hadoop.hbase.master.RegionManager: Skipped 0 region(s) that are in transition states
2009-06-02 11:02:21,164 DEBUG org.apache.hadoop.hbase.master.ServerManager: Process all wells: address: 192.168.1.87:62020, startcode: 1243954293575, load: (requests=57, regions=19, usedHeap=26, maxHeap=963) openingCount: 0, nobalancingCount: 4
2009-06-02 11:02:21,165 DEBUG org.apache.hadoop.hbase.master.RegionManager: Server is overloaded. Server load: 19 avg: 12.666666666666666, slop: 0.1
2009-06-02 11:02:21,165 DEBUG org.apache.hadoop.hbase.master.RegionManager: Choosing to reassign 6 regions. mostLoadedRegions has 10 regions in it.
{quote}
That's because the load is empty when adding a new region server so I added a check to instead use the load provided by the RS during the failover inspection. So it's fixed.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
> Attachments: hbase-1357-v1.patch, hbase-1357-v2.patch, hbase-1357-v3.patch, hbase-1357-v4.patch
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS (allowing 0.0.0.0) by having master actually write RS's address to ZK when it gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_ address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs. HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c the RS knows nothing about the master until the master address appears in ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful, its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.