You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jay Vyas <ja...@gmail.com> on 2013/08/05 16:52:28 UTC
Puppetized HBASE + Passwordless SSH examples
Hi everyone .
tl;dr? Any hints on where to start with puppetization of my hbase stack
would be much appreciated.
Here goes..
This is an open quesiton about puppet and hbase - any deployment stories or
suggested modules would be appreciated.
Have been looking around at some puppet modules for HBASE, and am what
experiences people might have had with some of them. In particular, for
HBase, you need modules for
- passwordless ssh
- hbase itself
- zookeeper
- hadoop mapred
- some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of the
time this is HDFS.
- firewall
So - I'm assuming since HBASE clusters are often dedicated, that somewhere
out there a mature end-to-end configuration template for puppetizing HBASE
against brand new servers is in existence, :)
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Puppetized HBASE + Passwordless SSH examples
Posted by Bryan Beaudreault <bb...@hubspot.com>.
A couple years ago we started our stack with this as a base:
http://hstack.org/hstack-automated-deployment-using-puppet/
However, we ended up simplifying it quite a bit. In the end, for us,
puppet just sets up the directories (i.e. data directories for hdfs,
namenode, etc), and installs the services from the right yum repo
(cloudera's or our own).
We then use fabric to push configs to the servers and for managing services.
We chose this approach for a few reasons:
1) We have multiple hbase and hadoop clusters with different
configurations, and managing those differences in puppet configs was a
headache.
2) Using manual pushing of configs over an automated approach (puppet)
gives you much greater control over the rollout of new configurations. We
found it annoying to have to stop puppet or otherwise gate certain changes
so we could test, for fear of puppet pushing it out everywhere.
3) Similar to how it is best practice to keep markup and presentation (html
and css) separate, we found it nice to keep configuration separate from
service installation/server setup.
We haven't opensourced our fabric-based configuration setup yet
unfortunately, but hopefully the above article and the idea of separating
the two may help.
On Mon, Aug 5, 2013 at 10:52 AM, Jay Vyas <ja...@gmail.com> wrote:
> Hi everyone .
>
> tl;dr? Any hints on where to start with puppetization of my hbase stack
> would be much appreciated.
>
> Here goes..
>
> This is an open quesiton about puppet and hbase - any deployment stories or
> suggested modules would be appreciated.
>
> Have been looking around at some puppet modules for HBASE, and am what
> experiences people might have had with some of them. In particular, for
> HBase, you need modules for
>
> - passwordless ssh
> - hbase itself
> - zookeeper
> - hadoop mapred
> - some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of the
> time this is HDFS.
> - firewall
>
> So - I'm assuming since HBASE clusters are often dedicated, that somewhere
> out there a mature end-to-end configuration template for puppetizing HBASE
> against brand new servers is in existence, :)
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Puppetized HBASE + Passwordless SSH examples
Posted by Asaf Mesika <as...@gmail.com>.
We have also developed a fabric customization for HBase stack and all our
components.
Anyone had experience with Ambari relative to Fabric?
On Tuesday, August 6, 2013, Jay Vyas wrote:
> thanks .. !
>
> well there certainly are a few customized ones on github but they are
> varying degrees of maintainance.
>
> i gather from this thread that no generic and accepted puppetization of
> hbase has been created in the community. maybe a jira ?
>
>
> On Tue, Aug 6, 2013 at 11:35 AM, Jeremy Carroll <phobos182@gmail.com<javascript:;>>
> wrote:
>
> > I have some modules that I developed for Pinterest if you are looking
> for a
> > head start. LMK if you would be interested.
> >
> >
> > On Tue, Aug 6, 2013 at 6:03 AM, Artem Ervits <are9004@nyp.org<javascript:;>>
> wrote:
> >
> > > Check out apache ambari, it uses Puppet to deploy and manage HBase and
> > > other apps.
> > >
> > >
> > > Artem Ervits
> > > Data Analyst
> > > New York Presbyterian Hospital
> > >
> > > ----- Original Message -----
> > > From: Jay Vyas [mailto:jayunit100@gmail.com <javascript:;>]
> > > Sent: Monday, August 05, 2013 10:52 AM
> > > To: user@hbase.apache.org <javascript:;> <user@hbase.apache.org<javascript:;>
> >
> > > Subject: Puppetized HBASE + Passwordless SSH examples
> > >
> > > Hi everyone .
> > >
> > > tl;dr? Any hints on where to start with puppetization of my hbase stack
> > > would be much appreciated.
> > >
> > > Here goes..
> > >
> > > This is an open quesiton about puppet and hbase - any deployment
> stories
> > or
> > > suggested modules would be appreciated.
> > >
> > > Have been looking around at some puppet modules for HBASE, and am what
> > > experiences people might have had with some of them. In particular,
> for
> > > HBase, you need modules for
> > >
> > > - passwordless ssh
> > > - hbase itself
> > > - zookeeper
> > > - hadoop mapred
> > > - some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of
> > the
> > > time this is HDFS.
> > > - firewall
> > >
> > > So - I'm assuming since HBASE clusters are often dedicated, that
> > somewhere
> > > out there a mature end-to-end configuration template for puppetizing
> > HBASE
> > > against brand new servers is in existence, :)
> > >
> > >
> > >
> > > --
> > > Jay Vyas
> > > http://jayunit100.blogspot.com
> > >
> > > This electronic message is intended to be for the use only of the named
> > > recipient, and may contain information that is confidential or
> > privileged.
> > > If you are not the intended recipient, you are hereby notified that
> any
> > > disclosure, copying, distribution or use of the contents of this
> message
> > is
> > > strictly prohibited. If you have received this message in error or are
> > not
> > > the named recipient, please notify us immediately by contacting the
> > sender
> > > at the electronic mail address noted above, and delete and destroy all
> > > copies of this message. Thank you.
> > >
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Puppetized HBASE + Passwordless SSH examples
Posted by Jay Vyas <ja...@gmail.com>.
thanks .. !
well there certainly are a few customized ones on github but they are
varying degrees of maintainance.
i gather from this thread that no generic and accepted puppetization of
hbase has been created in the community. maybe a jira ?
On Tue, Aug 6, 2013 at 11:35 AM, Jeremy Carroll <ph...@gmail.com> wrote:
> I have some modules that I developed for Pinterest if you are looking for a
> head start. LMK if you would be interested.
>
>
> On Tue, Aug 6, 2013 at 6:03 AM, Artem Ervits <ar...@nyp.org> wrote:
>
> > Check out apache ambari, it uses Puppet to deploy and manage HBase and
> > other apps.
> >
> >
> > Artem Ervits
> > Data Analyst
> > New York Presbyterian Hospital
> >
> > ----- Original Message -----
> > From: Jay Vyas [mailto:jayunit100@gmail.com]
> > Sent: Monday, August 05, 2013 10:52 AM
> > To: user@hbase.apache.org <us...@hbase.apache.org>
> > Subject: Puppetized HBASE + Passwordless SSH examples
> >
> > Hi everyone .
> >
> > tl;dr? Any hints on where to start with puppetization of my hbase stack
> > would be much appreciated.
> >
> > Here goes..
> >
> > This is an open quesiton about puppet and hbase - any deployment stories
> or
> > suggested modules would be appreciated.
> >
> > Have been looking around at some puppet modules for HBASE, and am what
> > experiences people might have had with some of them. In particular, for
> > HBase, you need modules for
> >
> > - passwordless ssh
> > - hbase itself
> > - zookeeper
> > - hadoop mapred
> > - some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of
> the
> > time this is HDFS.
> > - firewall
> >
> > So - I'm assuming since HBASE clusters are often dedicated, that
> somewhere
> > out there a mature end-to-end configuration template for puppetizing
> HBASE
> > against brand new servers is in existence, :)
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
> >
> > This electronic message is intended to be for the use only of the named
> > recipient, and may contain information that is confidential or
> privileged.
> > If you are not the intended recipient, you are hereby notified that any
> > disclosure, copying, distribution or use of the contents of this message
> is
> > strictly prohibited. If you have received this message in error or are
> not
> > the named recipient, please notify us immediately by contacting the
> sender
> > at the electronic mail address noted above, and delete and destroy all
> > copies of this message. Thank you.
> >
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Puppetized HBASE + Passwordless SSH examples
Posted by Jeremy Carroll <ph...@gmail.com>.
I have some modules that I developed for Pinterest if you are looking for a
head start. LMK if you would be interested.
On Tue, Aug 6, 2013 at 6:03 AM, Artem Ervits <ar...@nyp.org> wrote:
> Check out apache ambari, it uses Puppet to deploy and manage HBase and
> other apps.
>
>
> Artem Ervits
> Data Analyst
> New York Presbyterian Hospital
>
> ----- Original Message -----
> From: Jay Vyas [mailto:jayunit100@gmail.com]
> Sent: Monday, August 05, 2013 10:52 AM
> To: user@hbase.apache.org <us...@hbase.apache.org>
> Subject: Puppetized HBASE + Passwordless SSH examples
>
> Hi everyone .
>
> tl;dr? Any hints on where to start with puppetization of my hbase stack
> would be much appreciated.
>
> Here goes..
>
> This is an open quesiton about puppet and hbase - any deployment stories or
> suggested modules would be appreciated.
>
> Have been looking around at some puppet modules for HBASE, and am what
> experiences people might have had with some of them. In particular, for
> HBase, you need modules for
>
> - passwordless ssh
> - hbase itself
> - zookeeper
> - hadoop mapred
> - some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of the
> time this is HDFS.
> - firewall
>
> So - I'm assuming since HBASE clusters are often dedicated, that somewhere
> out there a mature end-to-end configuration template for puppetizing HBASE
> against brand new servers is in existence, :)
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
> This electronic message is intended to be for the use only of the named
> recipient, and may contain information that is confidential or privileged.
> If you are not the intended recipient, you are hereby notified that any
> disclosure, copying, distribution or use of the contents of this message is
> strictly prohibited. If you have received this message in error or are not
> the named recipient, please notify us immediately by contacting the sender
> at the electronic mail address noted above, and delete and destroy all
> copies of this message. Thank you.
>
Re: Puppetized HBASE + Passwordless SSH examples
Posted by Artem Ervits <ar...@nyp.org>.
Check out apache ambari, it uses Puppet to deploy and manage HBase and other apps.
Artem Ervits
Data Analyst
New York Presbyterian Hospital
----- Original Message -----
From: Jay Vyas [mailto:jayunit100@gmail.com]
Sent: Monday, August 05, 2013 10:52 AM
To: user@hbase.apache.org <us...@hbase.apache.org>
Subject: Puppetized HBASE + Passwordless SSH examples
Hi everyone .
tl;dr? Any hints on where to start with puppetization of my hbase stack
would be much appreciated.
Here goes..
This is an open quesiton about puppet and hbase - any deployment stories or
suggested modules would be appreciated.
Have been looking around at some puppet modules for HBASE, and am what
experiences people might have had with some of them. In particular, for
HBase, you need modules for
- passwordless ssh
- hbase itself
- zookeeper
- hadoop mapred
- some HCFS file system (HDFS, S3FileSystem, ...). Obviously, most of the
time this is HDFS.
- firewall
So - I'm assuming since HBASE clusters are often dedicated, that somewhere
out there a mature end-to-end configuration template for puppetizing HBASE
against brand new servers is in existence, :)
--
Jay Vyas
http://jayunit100.blogspot.com
This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.