Re: DevOps question : auto deployment/setup of Solr & Zookeeper on medium-large clusters

[Susheel Kumar <su...@gmail.com>]

Hi Davis,  I wanted to thank you for suggesting Ansible as one of the
automation tool as it has been working very well in automating the
deployments of Zookeeper, Solr on our clusters.

Thanks,
Susheel

On Wed, Oct 21, 2015 at 10:47 AM, Davis, Daniel (NIH/NLM) [C] <
daniel.davis@nih.gov> wrote:

> Susheel,
>
> Our puppet stuff is very close to our infrastructure, using specific
> Netapp volumes and such, and assuming some files come from NFS.
> It is also personally embarrassing to me that we still use NIS - doh!
>
> -----Original Message-----
> From: Susheel Kumar [mailto:susheel2777@gmail.com]
> Sent: Tuesday, October 20, 2015 8:34 PM
> To: solr-user@lucene.apache.org
> Subject: Re: DevOps question : auto deployment/setup of Solr & Zookeeper
> on medium-large clusters
>
> Thanks, Davis, Jeff.
>
> We are not using AWS.  Is there any scripts/framework already developed
> using puppet available?
>
> On Tue, Oct 20, 2015 at 7:59 PM, Jeff Wartes <jw...@whitepages.com>
> wrote:
>
> >
> > If you’re using AWS, there’s this:
> > https://github.com/LucidWorks/solr-scale-tk
> > If you’re using chef, there’s this:
> > https://github.com/vkhatri/chef-solrcloud
> >
> > (There are several other chef cookbooks for Solr out there, but this
> > is the only one I’m aware of that supports Solr 5.3.)
> >
> > For ZK, I’m less familiar, but if you’re using chef there’s this:
> > https://github.com/SimpleFinance/chef-zookeeper
> > And this might be handy to know about too:
> > https://github.com/Netflix/exhibitor/wiki
> >
> >
> > On 10/20/15, 6:37 AM, "Davis, Daniel (NIH/NLM) [C]"
> > <da...@nih.gov>
> > wrote:
> >
> > >Waste of money in my opinion.   I would point you towards other tools -
> > >bash scripts and free configuration managers such as puppet, chef, salt,
> > >or ansible.    Depending on what development you are doing, you may want
> > >a continuous integration environment.   For a small company starting
> out,
> > >using a free CI, maybe SaaS, is a good choice.   A professional version
> > >such as Bamboo, TeamCity, Jenkins are almost essential in a large
> > >enterprise if you are doing diverse builds.
> > >
> > >When you create a VM, you can generally specify a script to run after
> the
> > >VM is mostly created.   There is a protocol (PXE Boot) that enables this
> > >- a PXE server listens and hears that a new server with such-and-such
> > >Ethernet Address is starting.   The PXE server makes it boot like a
> > >CD-ROM/DVD install, booting from installation media on the network and
> > >installing.    Once that install is down, a custom script may be
> invoked.
> > >  This script is typically a bash script, because you may not be able to
> > >count on too much else being installed.   However, python/perl are also
> > >reasonable choices - just be careful that the modules/libraries you are
> > >using for the script are present.    The same PXE protocol is used in
> > >large on-premises installations (vCenter) and in the cloud
> > >(AWS/Digital Ocean).  We don't care about the PXE server - the point
> > >is that you can generally run a bash script after your install.
> > >
> > >The bash script can bootstrap other services such as puppet, chef, or
> > >salt, and/or setup keys so that push configuration management tools such
> > >as ansible can reach the server.   The bash script may even be smart
> > >enough to do all of the setup you need, depending on what other servers
> > >you need to configure.   Smart bash scripts are good for a small
> company,
> > >but for large setups, I'd use puppet, chef, salt, and/or ansible.
> > >
> > >What I tend to do is to deploy things in such a way that puppet
> > >(because it is what we use here) can setup things so that a "solradm"
> > >account can setup everything else, and solr and zookeeper are running
> as a "solrapp"
> > >user using puppet.    Then, my continuous integration server, which is
> > >Atlassian Bamboo (you can also use tools such as Jenkins, TeamCity,
> > >BuildBot), installs solr as "solradm" and sets it up to run as
> "solrapp".
> > >
> > >I am not a systems administrator, and I'm not really in "DevOps", my
> > >job is to be above all of that and do "systems architecture" which I
> > >am lucky still involves coding both in system administration and
> applications
> > >development.   So, that's my 2 cents.
> > >
> > >Dan Davis, Systems/Applications Architect (Contractor), Office of
> > >Computer and Communications Systems, National Library of Medicine,
> > >NIH
> > >
> > >-----Original Message-----
> > >From: Susheel Kumar [mailto:susheel2777@gmail.com]
> > >Sent: Tuesday, October 20, 2015 9:19 AM
> > >To: solr-user@lucene.apache.org
> > >Subject: DevOps question : auto deployment/setup of Solr & Zookeeper
> > >on medium-large clusters
> > >
> > >Hello,
> > >
> > >Resending to see opinion from Dev-Ops perspective on the tools for
> > >installing/deployment of Solr & ZK on large no of machines and
> > >maintaining them. I have heard Bladelogic or HP OO (commercial tools)
> > >etc. being used.
> > >Please share your experience or pros / cons of such tools.
> > >
> > >Thanks,
> > >Susheel
> > >
> > >On Mon, Oct 19, 2015 at 3:32 PM, Susheel Kumar
> > ><su...@gmail.com>
> > >wrote:
> > >
> > >> Hi,
> > >>
> > >> I am trying to find the best practises for setting up Solr on new
> > >> 20+ machines  & ZK (5+) and repeating same on other environments.
> > >> What's the best way to download, extract, setup Solr & ZK in an
> > >> automated way along with other dependencies like java etc.  Among
> > >> shell scripts or puppet or docker or imaged vm's what is being used
> > >> & suggested from Dev-Ops perspective.
> > >>
> > >> Thanks,
> > >> Susheel
> > >>
> >
> >
>