You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by sblackmon <sb...@apache.org> on 2016/10/05 16:00:07 UTC

persister integration testing refactoring/improvement exemplar

All, 

I’ve pushed a branch that depicts the integration testing with databases in docker that I envision adopting across persisters and examples this quarter.

https://github.com/apache/incubator-streams/compare/master...steveblackmon:STREAMS-400?expand=1

Fully resolves:

STREAMS-400
streams-persist-elasticsearch : bump to version 2.x

STREAMS-404
Integration Test for FsElasticsearchIndex

The highlight though is that making docker a standard part of the build environment, we can 
a) toss out the hodgepodge of maven plugins that startup databases for testing
b) avoid relying on integration test scaffolds maintained by database SDK libraries
c) make it much easier to ensure compatibility with the most current database versions (just run the test suite using ‘latest’ tags)
d) reduce the delta between how the readers/writers operate during the test vs in the wild
e) other than the initial pull of the docker image these type of tests seem to run much faster with the DB in a separate container

The only downside is that to run these integration tests devs need a docker environment, but I don’t see that as a significant problem.

The other nice thing about this change-set is how every IT has fully sufficient and re-usable src/test/resource .conf file rather than managing test configurations within the code itself.

There’s a few things to figure out regarding how to best tie the docker plugin up with the maven reactor.  Ideally we want

mvn -PdockerITs verify -DskipITs=false 

to work from every level of the reactor.  In this branch it only works when run directly from streams-persist-elasticsearch

NOTE: to run this locally you’ll need to mvn install
https://github.com/steveblackmon/incubator-streams-master/tree/STREAMS-400

Please let me know what you think.  If no one objects I’ll create tickets to apply this pattern to mongo, kafka, neo4j, hbase across streams-project/streams-contrib/streams-persist-* and streams-examples/*  

Steve