Re: HBase Integration Tests running on bigtop01.cloudera.org?

[Stack <st...@duboce.net>]

(Adding dev@hbase to the conversation -- below is follow up to request
asking if we could run hbase it tests on bigtop infrastructure)

Let me add more to Andrew's answers.

On Fri, Dec 21, 2012 at 12:02 PM, Andrew Purtell <ap...@apache.org>wrote:

> Hi Roman and Stack,
>
> >  let us know if your system tests can be deployed to Maven with every
> nightly build of HBase as a standalone artifact that Bigtop's iTest can
> simply depend on
>
> I believe we have an "hbase-it" artifact containing integration tests that
> pulls in all of the other HBase artifacts.
>
>
What Andrew says.  It has a few tests already:

http://svn.apache.org/viewvc/hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/

One of these in particular, IntegrationTestDataIngestWithChaosMonkey, would
be sweet to run on a period.

You want us to push a nightly snapshot up to apache maven repo?  We'd just
have apache jenkins do that?



> >  let us know what's your sweet spot for the # of nodes/RS
>
> One of our large tests -- TestReplication -- is going to be moved out into
> an integration test, and this will want to configure two independent
> clusters with client access to both. Assume two 3 slave clusters, that's a
> total of 8. Otherwise for single cluster tests 5 slaves seems a good
> starting point IMO.
>
>
The tests in hbase-it can are sized.  If the cluster is bigger, we can up
the test parameters to flood it.  5 is probably a good size to start with.

Where would the configs go?  Would we have to give these up front or is
there means, somewhere where we can commit, of managing them ourselves?



> > what are you expectations for tests that do manipulate the state of the
> cluster (like ChaosMonkey) -- do you expect unrestricted ssh, etc?
>
> I think we want, at least, unrestricted access on the service accounts for
> hdfs, hbase, zookeeper, so we can kill -9 or kill -STOP processes at
> different system layers.
>
>
What Andrew said.  Minimally we'd want access to the logs (or at least have
them collected at test end and made accessible somewhere).



> > All in all, I think this makes perfect sense and if there are folks in
> the HBase community willing to help us get this going it'll be a 'beginning
> of a beautiful friendship' ;-)
>
> Yes. Great!
>
>
Chalk me down as someone willing to help get the 'beautiful friendship' off
the ground (Others wil be interested too....)

St.Ack



>
> On Fri, Dec 21, 2012 at 11:44 AM, Roman Shaposhnik <rv...@apache.org> wrote:
>
> > Hi Stack!
> >
> > Let me know at which point this discussion needs to be CCed
> > over to dev@hbase, but I'll just reply here for now:
> >
> > On Fri, Dec 21, 2012 at 10:58 AM, Stack <st...@duboce.net> wrote:
> > > The HBase Project has been developing a set of big tests that are too
> > large
> > > to run as part of the regular hbase project unit tests suite.  They are
> > > currently run adhoc by individual developers.  We are looking for a
> home
> > > for these tests, a location where they could be automatically run with
> > some
> > > periodicity.
> > >
> > > Would it be possible to run them on the bigtop test infrastructure?
> >
> > I think Bigtop makes a perfect home for these tests. After all, Bigtop
> > as a project is exactly about facilitating integration efforts of Hadoop
> > and its ecosystem projects.
> >
> > > Their general nature is set up a cluster, run a number of pretty
> > intensive
> > > steps -- e.g. loading lots of artificial data -- while optionally, a
> > chaos
> > > monkey is doing its damndest to break things.   The tests can be sized
> to
> > > match cluster size, from mini-all-in-the-one-jvm up to clusters of
> > hundreds
> > > of nodes.   On the end, the test generates a report and logs.  A fuller
> > > description can be found here [1].
> > >
> > > What you all think?
> >
> > Bigtop infrastructure can offer the following (provided that there's
> > interest
> > in HBase community to help with keeping an eye on things since Bigtop
> > is as short on cycles as any other ASF project):
> >    * regular builds of Bigtop's trunk and thus a particular point
> >      of HBase tree that Bigtop's trunk is tracking (on all supported
> >      Linux platforms)
> >    * regular builds of HBase's trunk (on all supported Linux platforms)
> >    * regular test runs on dynamically deployed Hadoop clusters. We
> >      are currently deploying to ~5 nodes which gives us total HDFS
> >      capacity of around couple of T.
> >    * jenkins reporting of those test runs
> >
> > Here's what you guys can do in order to make the job of hooking up
> > your tests easier:
> >    * let us know what's your sweet spot for the # of nodes/RS
> >    * let us know what's your sweet spot for the total storage capacity
> >    * let us know if your system tests can be deployed to Maven
> >      with every nightly build of HBase as a standalone artifact that
> >      Bigtop's iTest can simply depend on
> >    * provide a bit more details on the desired configs for the tests
> >    * what are you expectations for tests that do manipulate the
> >      state of the cluster (like ChaosMonkey) -- do you expect
> >      unrestricted ssh, etc?
> >
> > All in all, I think this makes perfect sense and if there are folks
> > in the HBase community willing to help us get this going it'll
> > be a 'beginning of a beautiful friendship' ;-)
> >
> > Thanks,
> > Roman.
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>