You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/01/21 20:07:59 UTC
[jira] Created: (HBASE-1147) Modify the scripts to use Zookeeper
Modify the scripts to use Zookeeper
-----------------------------------
Key: HBASE-1147
URL: https://issues.apache.org/jira/browse/HBASE-1147
Project: Hadoop HBase
Issue Type: Sub-task
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667528#action_12667528 ]
Nitay Joffe commented on HBASE-1147:
------------------------------------
Looks good.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147-v2.patch, hbase-1147-v3.patch, hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667332#action_12667332 ]
stack commented on HBASE-1147:
------------------------------
bq. I think I can refactor to make it configurable.
Not important
bq. Yes, it is a common command. Probably should use the full name tho (netcat).
You might try and verify its available on other OSs and that the parameters are same.
For example, up on hudson:
{code}
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
-bash-3.00$ nc
-bash: nc: command not found
-bash-3.00$ netcat
-bash: netcat: command not found
-bash-3.00$ which netcat
no netcat in /opt/sfw/bin /opt/subversion-1.4.5/bin /usr/bin
{code}
What happens if you kill the zk process? Does it run shutdown handler?
bq. Not sure I understand you but...
I think you got what I was saying. Was just asking if STDOUT went into a .out file or not. Sounds like it does.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667348#action_12667348 ]
Nitay Joffe commented on HBASE-1147:
------------------------------------
Please see my comments in 1144. I think our configuration scheme is too complicated.
https://issues.apache.org/jira/browse/HBASE-1144?focusedCommentId=12667125#action_12667125
IRC conversation with Stack about this:
Jan 26 10:20:21 <St^Ack> So, whats this about:
Jan 26 10:20:22 <nitay> which is really just ripping code out of ZK's repo
Jan 26 10:20:32 <St^Ack> The list of quorum servers specified in <code>${HBASE_HOME}/conf/zoo.cfg</code> should also be written to <code>${HBASE_HOME}/conf/zookeepers</code>.
Jan 26 10:20:39 <St^Ack> "The list of quorum servers specified in <code>${HBASE_HOME}/conf/zoo.cfg</code> should also be written to <code>${HBASE_HOME}/conf/zookeepers</code>.
Jan 26 10:20:43 <St^Ack> .."
Jan 26 10:20:49 <St^Ack> info in two places?
Jan 26 10:20:55 <nitay> three actually
Jan 26 10:21:05 <nitay> that's why i wrote in my comment we should fix that
Jan 26 10:21:07 * idefine (i=root@twizzler.dreamhost.com) has joined #hbase
Jan 26 10:21:13 <nitay> so zoo.cfg is purely for ZK
Jan 26 10:21:16 * St^Ack HHHRNHH!!!! wrong answer
Jan 26 10:21:29 <nitay> well its a part of 1147
Jan 26 10:21:39 <nitay> i was commenting on it saying we should discuss it
Jan 26 10:21:49 <nitay> zookeepers is for HBase, its analogous to regionservers file
Jan 26 10:22:06 * St^Ack looking
Jan 26 10:22:07 <nitay> and then there's the option in hbase-site.xml
Jan 26 10:22:12 <nitay> read my whole comments
Jan 26 10:22:34 <nitay> specifically, https://issues.apache.org/jira/browse/HBASE-1144?focusedCommentId=12667125#action_12667125
Jan 26 10:23:05 <St^Ack> oh, you are on it already
Jan 26 10:23:11 <St^Ack> sorry missed that
Jan 26 10:23:16 <idefine> Hi all, quick question (new to hadoop/hbase). If I have a bunch of data in a database, do I need to write that data to disk inorder to use hadoop to map/reduce? or can the map function pull from the database by offsetting that starting index?
Jan 26 10:23:37 <idefine> the starting index*
Jan 26 10:23:49 <nitay> so most of that is part of 1147, i wrote the overview.html to match the current way its done, but i think really we should just have zoo.cfg, and parse everything from there
Jan 26 10:26:34 <St^Ack> nitay: just commented. you were on the ball. i was looking at the cheerleaders
Jan 26 10:27:28 <St^Ack> idefine: I believe there db hookups if you go looking through hadoop; if not in hadoop itself or in contrib, then look in JIRAs for a patch
Jan 26 10:27:38 <nitay> St^Ack, by populated u mean the whole host:port goodness? cause i was just thinking of a true/false flag
Jan 26 10:27:57 <nitay> if true, parse clientPort= and server.X= out of zoo.cfg and start/stop with Hbase
Jan 26 10:28:09 <nitay> if not, still have to parse those, but dont try to start/stop
Jan 26 10:28:22 <St^Ack> idefine: my faint knowledge comes of quick peruse of hadoop mailing list subject lines so don't put to much weight on it
Jan 26 10:29:03 <St^Ack> nitay: how do hbase clients know which remote zk cluster?
Jan 26 10:29:34 <idefine> St^Ack: what if I am using hbase for my database, would it make a difference?
Jan 26 10:30:09 <St^Ack> idefine: in that case, there are existant hookups. See under the hbase mapred package
Jan 26 10:30:19 * St^Ack goes to reread nitay comment
Jan 26 10:30:24 <idefine> St^Ack: thanks, will do.
Jan 26 10:30:40 <nitay> St^Ack, oh b/c hbase clients have just the HBaseConfiguration thing in the code u're saying, no zoo.cfg?
Jan 26 10:31:37 <St^Ack> nitay: no, zoo.cfg will be on classpath if its in conf dir.
Jan 26 10:31:47 <nitay> St^Ack, right ye so im confused
Jan 26 10:32:00 <nitay> why u need to ever specify ZK host:port in hbase-X.xml?
Jan 26 10:32:01 <St^Ack> nitay: so, if a remote cluster, zoo.cfg will contain the addresses of the remote zk quorum?
Jan 26 10:32:06 <nitay> yes
Jan 26 10:32:10 <St^Ack> ok.
Jan 26 10:32:13 <nitay> it has to, that's how zookeeper works
Jan 26 10:32:18 <St^Ack> then boolean as you suggest is way to go.
Jan 26 10:32:25 <nitay> zoo.cfg:
Jan 26 10:32:25 <nitay> tickTime=2000
Jan 26 10:32:25 <nitay> dataDir=/var/zookeeper
Jan 26 10:32:25 <nitay> clientPort=2181
Jan 26 10:32:25 <nitay> initLimit=5
Jan 26 10:32:25 <nitay> syncLimit=2
Jan 26 10:32:25 <nitay> server.1=zoo1:2888:3888
Jan 26 10:32:25 <nitay> server.2=zoo2:2888:3888
Jan 26 10:32:25 <nitay> server.3=zoo3:2888:3888
Jan 26 10:32:49 <nitay> so if u dont have any server.X lines its a local instance
Jan 26 10:33:01 <St^Ack> I didn't realize that clients would read zoo.conf to get to remote quorum also.
Jan 26 10:33:34 <nitay> right i didnt think about that at first too, but then i realized ll the info we need is in ZK's zoo.conf, untouched by us, just in different forms
Jan 26 10:33:46 <nitay> i also think its important we never add stuff to zoo.cfg, keep it exactly the same as ZK's
Jan 26 10:34:23 <St^Ack> agreed
Jan 26 10:34:28 <St^Ack> updated my comment in issue
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1147:
--------------------------------------
Attachment: hbase-1147-v3.patch
Fixes a comment in zoo.cfg, defaults HBASE_MANAGES_ZK to "true", removes useless config passed in start/stop-hbase.sh for ZK.
Another issue will be opened to fix the fact that dataDir will have to be configured in hbase-site and not zoo.cfg.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147-v2.patch, hbase-1147-v3.patch, hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667594#action_12667594 ]
stack commented on HBASE-1147:
------------------------------
+1
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147-v2.patch, hbase-1147-v3.patch, hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667268#action_12667268 ]
Jean-Daniel Cryans commented on HBASE-1147:
-------------------------------------------
bq. I'm guessing zk expects the config file to be named zoo.cfg
I think I can refactor to make it configurable.
bq. Whats this: '+ echo kill | nc 127.0.0.1 2181' Is it platform independent (at least to some degree? Does nc work same on sunos, bsd, and linux?).
Yes, it is a common command. Probably should use the full name tho (netcat).
bq. Is it not possible to change the zk port number in its zoo.cfg? So will 2181 always work? Or 127.0.0.1 even?
Nah I should probably grep.
bq. Does zk emit logging on STDOUT? What if it OOMEs? Is that caught somewhere or does it come out on STDOUT? Does start/stop zk not fit the hbase-daemon.sh mold? If you could shoehorn it in there, could drop PIDs so could kill these shutting it down rather than use nc and logs would be set up for you? (I'm guessing it just does't fit the mold).
Not sure I understand you but... the current situation is that hbase-zookeeper.sh calls hbase-daemon.sh on all machines in conf/zookeepers so the logs are actually setup.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667349#action_12667349 ]
Nitay Joffe commented on HBASE-1147:
------------------------------------
Your scripts do not seem to write ${dataDir}/myid in a replicated case. That is say I have a zoo.cfg like:
tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
There needs to be a file called myid with the X in server.X written to /var/zookeeper on each node. Is the user currently required to do this on their own?
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1147:
--------------------------------------
Attachment: hbase-1147.patch
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666752#action_12666752 ]
stack commented on HBASE-1147:
------------------------------
I'm guessing zk expects the config file to be named zoo.cfg?
Whats this: '+ echo kill | nc 127.0.0.1 2181' Is it platform independent (at least to some degree? Does nc work same on sunos, bsd, and linux?). Is it not possible to change the zk port number in its zoo.cfg? So will 2181 always work? Or 127.0.0.1 even? Suggest making these configs.... in hbase-env.sh if you have to or grep them from zoo.cfg.
nc looks sweet.
Does zk emit logging on STDOUT? What if it OOMEs? Is that caught somewhere or does it come out on STDOUT? Does start/stop zk not fit the hbase-daemon.sh mold? If you could shoehorn it in there, could drop PIDs so could kill these shutting it down rather than use nc and logs would be set up for you? (I'm guessing it just does't fit the mold).
Make copyright 2009 on new files.
Otherwise, patch looks good to me.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1147:
--------------------------------------
Attachment: hbase-1147-v2.patch
Grossly simplified version of the scripts. A setting in hbase-env tells whether HBase should manage a local instance of ZK or not. If it does manage it, the ZK server will be started on localhost on port 2181.
Also it no longer uses nc.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147-v2.patch, hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1147:
--------------------------------------
Attachment: (was: hbase-1147.patch)
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1147.
---------------------------------------
Resolution: Fixed
Committed to trunk.
To those who follow trunk closely, be advise that the start-hbase.sh script now starts a ZK instance unless HBASE_MANAGES_ZK in hbase-env.sh is set to false.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147-v2.patch, hbase-1147-v3.patch, hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1147) Modify the scripts to use Zookeeper
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1147:
--------------------------------------
Attachment: hbase-1147.patch
Request for review.
> Modify the scripts to use Zookeeper
> -----------------------------------
>
> Key: HBASE-1147
> URL: https://issues.apache.org/jira/browse/HBASE-1147
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Attachments: hbase-1147.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.