You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Bruno Dumon (JIRA)" <ji...@apache.org> on 2011/07/15 16:54:59 UTC
[jira] [Created] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Allow to specify hbase-site.xml properties through cluster configuration file
-----------------------------------------------------------------------------
Key: WHIRR-339
URL: https://issues.apache.org/jira/browse/WHIRR-339
Project: Whirr
Issue Type: Improvement
Components: service/hbase
Reporter: Bruno Dumon
Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Bruno Dumon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruno Dumon updated WHIRR-339:
------------------------------
Attachment: WHIRR-339.patch
Adding patch, corresponds to the changes at https://github.com/bdumon/whirr/commit/948b3aa08c062b24f5c154575cfac451a7cefdad
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Attachments: WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Bruno Dumon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruno Dumon updated WHIRR-339:
------------------------------
Attachment: WHIRR-339.patch
Updated the patch for current trunk. Ran HBase & CDH integration tests successfully.
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Fix For: 0.6.0
>
> Attachments: WHIRR-339.patch, WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu resolved WHIRR-339.
-------------------------------
Resolution: Fixed
Assignee: Bruno Dumon
I've just committed this. Thanks Bruno!
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Assignee: Bruno Dumon
> Fix For: 0.6.0
>
> Attachments: WHIRR-339.patch, WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066114#comment-13066114 ]
Tom White commented on WHIRR-339:
---------------------------------
This looks good - thanks for working on it.
> I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
hbase-site.hbase.client.retries.number was increased from the default due to problems we were having as described in WHIRR-314. So I think we should keep it in the absence of a better solution.
Have you managed to run the HBase integration tests with this change?
> Solving this is a topic of its own.
Worth opening a JIRA for this?
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Attachments: WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080676#comment-13080676 ]
Andrei Savu commented on WHIRR-339:
-----------------------------------
+1 HBase integration tests also pass on cloudservers.
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Fix For: 0.6.0
>
> Attachments: WHIRR-339.patch, WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Bruno Dumon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066917#comment-13066917 ]
Bruno Dumon commented on WHIRR-339:
-----------------------------------
I created WHIRR-342 for the issue regarding configuration/role conflicts.
Meanwhile learned that the patch as-is will break the hbase testcases (because of WHIRR-342), which run a thrift server on a separate node without master/regionserver. The instance-templates in whirr-hbase-test.properties (for both plain/cdh hbase) should be adjusted. The patches on WHIRR-240 & WHIRR-334 actually contain such a change already, so with those applied all should be ok.
Regarding hbase.client.retries.number: ok, interesting, we should include it then.
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Attachments: WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-339) Allow to specify hbase-site.xml
properties through cluster configuration file
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-339:
------------------------------
Fix Version/s: 0.6.0
> Allow to specify hbase-site.xml properties through cluster configuration file
> -----------------------------------------------------------------------------
>
> Key: WHIRR-339
> URL: https://issues.apache.org/jira/browse/WHIRR-339
> Project: Whirr
> Issue Type: Improvement
> Components: service/hbase
> Reporter: Bruno Dumon
> Fix For: 0.6.0
>
> Attachments: WHIRR-339.patch
>
>
> Currently the hbase-site.xml configuration is hardcoded in the configure(_cdh)_hbase.sh script. This patch adds dynamic generation of hbase-site.xml, modelled after what is done for hadoop-(common|mapred|hdfs)-site.xml.
> The user is able to specify custom properties in their cluster configuration, by prefixing them with 'hbase-site.'
> Default config is in whirr-hbase-default.properties. Compared to what was in configure_hbase.sh before, I dropped the setting of "hbase-site.hbase.client.retries.number=100" because I'm not sure we should divert from the HBase default there.
> I left in the passing of the ZOOKEEPER_QUORUM to the configure script, as I figured people using custom scripts might still want to have it (there's a PORT variable passed that doesn't seem to be used either).
> On a more general note, this patch has the same limitations as for Hadoop: running master and regionserver on same node will not work as the configuration will be appended twice to the same file, and running the restserver/thriftserver not together with regionserver or master will fail due to missing hbase-site.xml. Solving this is a topic of its own.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira