You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Lars George (JIRA)" <ji...@apache.org> on 2010/12/25 12:31:47 UTC
[jira] Created: (WHIRR-177) [HBase] Support embedded ZooKeeper
[HBase] Support embedded ZooKeeper
----------------------------------
Key: WHIRR-177
URL: https://issues.apache.org/jira/browse/WHIRR-177
Project: Whirr
Issue Type: Improvement
Affects Versions: 0.3.0
Reporter: Lars George
Assignee: Lars George
Fix For: 0.3.0
This can be achieved like so:
* Check if there is no "zk" role in the template -> set HBaseManagesZK = true
* Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
* Set the option in hbase-env.sh to indicate that HBase manages ZK:
{code}HBASE_MANAGES_ZK=true{code}
Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975182#action_12975182 ]
Lars George commented on WHIRR-177:
-----------------------------------
I am actually planning to add an implicit single embedded zookeeper node when nothing is specified bit also adding a "hbase-zookeeper" role that you can use in the template definition to explicitly specify which HBase node is to spin up a ZooKeeper node as well. That covers all scenarios. Does that make sense?
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
> Fix For: 0.3.0
>
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975147#action_12975147 ]
Andrew Purtell commented on WHIRR-177:
--------------------------------------
Thinking when hacking the bash scripts was the cluster is not HA (no multimaster, no clustered namenode, etc.) so you can just colocate a single ZK instance on the HBase master node. If there were HA features, then it might make sense to choose a subset of slaves at random, on some kind of log scale: ~10 slaves = 3 ZK nodes, ~100 slaves = 5 ZK nodes, ~1000 slaves = 7 or 9 ZK nodes.
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
> Fix For: 0.3.0
>
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992926#comment-12992926 ]
Lars George commented on WHIRR-177:
-----------------------------------
When using the non-implicit start of ZK on the master but supporting "hbase-zookeeper" then we would also have to make sure we create the proper "servers" section and myid's for all ZK peers.
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
> Fix For: 0.4.0
>
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993139#comment-12993139 ]
Tom White commented on WHIRR-177:
---------------------------------
It might be easier to generate the configuration files on the client, using an approach like WHIRR-55.
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
> Fix For: 0.4.0
>
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-177:
------------------------------
Fix Version/s: (was: 0.4.0)
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-177) [HBase] Support embedded ZooKeeper
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992925#comment-12992925 ]
Lars George commented on WHIRR-177:
-----------------------------------
To enable this we need to add the ZooKeeper details to the hbase-site.xml and start it through the HBase daemon script, for example:
{code}
HBASE_HOME=`ls -d /usr/local/hbase-*`
# HBase configuration (Zookeeper)
cat > $HBASE_HOME/conf/hbase-site.xml <<EOF
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>$ZOOKEEPER_QUORUM</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/mnt/hbase/zk</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>100</value>
</property>
</configuration>
EOF
...
mkdir -p /mnt/hbase/logs
mkdir -p /mnt/hbase/zk
...
[ ! -f /etc/hosts ] && echo "127.0.0.1 localhost" > /etc/hosts
...
"$HBASE_HOME"/bin/hbase-daemon.sh start zookeeper
{code}
Obviously we need to make sure not to override any existing hbase-site.xml but add to it.
> [HBase] Support embedded ZooKeeper
> ----------------------------------
>
> Key: WHIRR-177
> URL: https://issues.apache.org/jira/browse/WHIRR-177
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Affects Versions: 0.3.0
> Reporter: Lars George
> Assignee: Lars George
> Fix For: 0.4.0
>
>
> This can be achieved like so:
> * Check if there is no "zk" role in the template -> set HBaseManagesZK = true
> * Emit a zoo.cfg from the HBase post-configure script next to the hbase-site.xml
> * Set the option in hbase-env.sh to indicate that HBase manages ZK:
> {code}HBASE_MANAGES_ZK=true{code}
> Question is how many nodes run ZK? All slaves? A subset?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira