Unit/Integration tests with Mini Accumulo Cluster

[Jim Hughes <jh...@ccri.com>]

Hi all,

I work on GeoMesa and for the Accumulo 1.x line, we have been using the 
MockAccumulo infrastructure for our unit/integration tests which run in 
a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're 
looking at using the MiniAccumulo cluster infrastructure.

Are there best practices or examples of how to best use and reuse such a 
cluster between unit tests?

I see two broad options:  First, using Maven's integration test 
framework to start a cluster 'outside' of a unit test.  Second, we've 
build a test runner for HBase which starts a cluster and then runs the 
various tests.  This second approach has some small downsides, and 
that's why I'm asking if the you all have any great ideas!

Cheers,

Jim

Re: Unit/Integration tests with Mini Accumulo Cluster

[Jonathan Wonders <jw...@gmail.com>]

Jim,

I've found that sharing a minicluster instance via a lazily-initialized
static singleton is simple and effective within tests.  The reason I lean
toward this over the maven plugin is because it works both when executing
tests as part of a maven build and when executing tests from an IDE.  Also,
iterator and other server side code is easy to test with this approach.
Most unit testing frameworks do not provide lifecycle hooks for
initializing before all tests and tearing down after all tests, so the
options are limited for tearing down the minicluster.  Shutdown hooks are
probably the best option.

Here is a basic example that is integrated with JUnit 5 parameter resolvers.
https://github.com/jwonders/ax-accumulo/tree/master/src/test/java/com/jwsphere/accumulo/async

It would be straightforward to extend this to use the maven plugin in the
context of a maven build by choosing a provider implementation based on a
system property as Keith has suggested.  I haven't tried this myself so I'm
not sure exactly how to deal with modules that build server-side code.  The
jar should be available when the minicluster is started by the maven plugin
but I don't know the details of when maven resolves the plugin dependencies
to a file.

Of course, sharing a minicluster means you have to be careful to make tests
independent.  In my experience, it has been worthwhile.



On Fri, Dec 13, 2019 at 1:49 PM Keith Turner <ke...@deenlo.com> wrote:

> One more point about Fluo.  ITBase relies on maven setting a property.
> That is done by the following link.
>
>
> https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/pom.xml#L333
>
> On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes <jh...@ccri.com> wrote:
> >
> > Hi all,
> >
> > I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> > MockAccumulo infrastructure for our unit/integration tests which run in
> > a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> > looking at using the MiniAccumulo cluster infrastructure.
> >
> > Are there best practices or examples of how to best use and reuse such a
> > cluster between unit tests?
> >
> > I see two broad options:  First, using Maven's integration test
> > framework to start a cluster 'outside' of a unit test.  Second, we've
> > build a test runner for HBase which starts a cluster and then runs the
> > various tests.  This second approach has some small downsides, and
> > that's why I'm asking if the you all have any great ideas!
> >
> > Cheers,
> >
> > Jim
> >
>

Re: Unit/Integration tests with Mini Accumulo Cluster

[Keith Turner <ke...@deenlo.com>]

One more point about Fluo.  ITBase relies on maven setting a property.
That is done by the following link.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/pom.xml#L333

On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes <jh...@ccri.com> wrote:
>
> Hi all,
>
> I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> MockAccumulo infrastructure for our unit/integration tests which run in
> a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> looking at using the MiniAccumulo cluster infrastructure.
>
> Are there best practices or examples of how to best use and reuse such a
> cluster between unit tests?
>
> I see two broad options:  First, using Maven's integration test
> framework to start a cluster 'outside' of a unit test.  Second, we've
> build a test runner for HBase which starts a cluster and then runs the
> various tests.  This second approach has some small downsides, and
> that's why I'm asking if the you all have any great ideas!
>
> Cheers,
>
> Jim
>

Re: Unit/Integration tests with Mini Accumulo Cluster

[Mike Miller <mm...@apache.org>]

The Accumulo ITs (any java files whose name ends in IT) would be a good
example since they are all run using MiniAccumuloCluster, with some being
able to run against a standalone cluster[1].  Some ITs start and stop a
cluster for every test, while others keep a cluster running for the whole
IT.  Any ITs that extend the AccumuloClusterHarness are standalone
capable.  For more information how to run the ITs against a standalone
cluster see the TESTING.md[2].

[1]
https://github.com/apache/accumulo/tree/3fd5cad92f9b63ac19e4466f3f2d5237b905262c/test/src/main/java/org/apache/accumulo/test
[2] https://github.com/apache/accumulo/blob/master/TESTING.md

On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes <jh...@ccri.com> wrote:

> Hi all,
>
> I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> MockAccumulo infrastructure for our unit/integration tests which run in
> a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> looking at using the MiniAccumulo cluster infrastructure.
>
> Are there best practices or examples of how to best use and reuse such a
> cluster between unit tests?
>
> I see two broad options:  First, using Maven's integration test
> framework to start a cluster 'outside' of a unit test.  Second, we've
> build a test runner for HBase which starts a cluster and then runs the
> various tests.  This second approach has some small downsides, and
> that's why I'm asking if the you all have any great ideas!
>
> Cheers,
>
> Jim
>
>

Re: Unit/Integration tests with Mini Accumulo Cluster

[Keith Turner <ke...@deenlo.com>]

You could use the Accumulo Maven Plugin which starts an Accumulo
cluster for using MiniAccumulo.

https://accumulo.apache.org/accumulo2-maven-plugin/

Fluo, uses this and below are a few pointers.

The following is a base for Fluo ITs that uses the mini Accumulo
started by the maven plugin if it exists, otherwise it starts a mini
(useful for running in an IDE).

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/modules/integration-tests/src/main/java/org/apache/fluo/integration/ITBase.java#L74

Fluo is a multi module project.  The link below shows configuring the
plugin in the parent pom.  Plugins do not inherit the dependencies of
the project, that is why the plugin explicitly configures the
Accumulo, Hadoop, and ZK versions.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/pom.xml#L279

The link below shows activating the plugin in the IT module.  It
inherits the config from the module.  Not all modules activate the
plugin.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/modules/integration-tests/pom.xml#L103

On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes <jh...@ccri.com> wrote:
>
> Hi all,
>
> I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> MockAccumulo infrastructure for our unit/integration tests which run in
> a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> looking at using the MiniAccumulo cluster infrastructure.
>
> Are there best practices or examples of how to best use and reuse such a
> cluster between unit tests?
>
> I see two broad options:  First, using Maven's integration test
> framework to start a cluster 'outside' of a unit test.  Second, we've
> build a test runner for HBase which starts a cluster and then runs the
> various tests.  This second approach has some small downsides, and
> that's why I'm asking if the you all have any great ideas!
>
> Cheers,
>
> Jim
>