You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by N Keywal <nk...@gmail.com> on 2011/10/20 22:46:00 UTC
[unit tests] small tests vs. others
Hi,
I've done a stat on the tests we have. Many of them are actually quite
simple: There are 226 test classes in HBase. 100 of them does not use a
cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
but need between 2 and 40 seconds.
I tried to execute these with different maven settings.
fork: once or always, i.e. all the tests are executed in the same JVM or
there is a new JVM created for every test class
parallel; deactivated or activated, with 3 threads and a per class
parallelisation.
The 100 tests executes for the whole maven stuff, including the initial
dependency & task analysis
fork always: 1m50s
fork once: 30 seconds
parallel: 30s (=> no difference).
It works quite well, except with TestLruBlockCache that I had to remove.
I tried as well with the 24 'bigger' ones but no cluster, for the following
results:
fork always: 7m36s
parallel: 2m32s: 3 times faster.
However, I had to remove many of them, there are some conflicts that I have
not solved (nor tried to).
The fact that nearly 50% of the tests can be executed in 1 minutes is
interesting imho.
That makes me wonder if we should try to get a separation of the tests in 3
parts, small/medium/large as of
http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
small would be:
- less than two seconds, can be executed in the same jvm
- all of them: less than 2 minute
- no cluster, nothing that can break the environment
- run by the developers as a part of the build
medium:
- less than 40 seconds
- all of them: would last less than 30 minutes, ideally 15 minutes
- run in an isolated jvm
- run explicitly by the developers before submitting a fix
large:
- run in an isolated jvm
- Should last less than 45minutes
- run on the central built or explicitly by the developers
- break the built when it fails.
Only one issue: it's not directly possible with maven, but there are known
workarounds :-)
What do you think?
Below the tests set for the 120 tests that seems to work altogether.
mvn test
-Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
Re: [unit tests] small tests vs. others
Posted by N Keywal <nk...@gmail.com>.
That sounds perfect :-).
For JUnit, there is nothing specific I used the script provided with JUnit
(build_tests.sh).
For surefire, I had to use -DskipTests to get it installed, even before my
changes. Except this (!) it's pure standard maven.
Then there are two use cases:
- short term, we just need to change the versions and the repository list
in the .pom to see how it behaves.
- medium term (3/4 days) we should publish the category split, I created
4781 to put the .pom I am currently testing.
Thanks for your help!
On Mon, Nov 14, 2011 at 8:47 PM, Gary Helmling <gh...@gmail.com> wrote:
> Hi N,
>
> I can push up the modified surefire and junit build with appropriate
> versions.
>
> Is there anything special to either build that would complicate
> building with the supplied patches? If not, then I can do the builds
> and push up the artifacts. Then we coordinate the testing. How does
> that sound?
>
> --gh
>
>
> On Mon, Nov 14, 2011 at 5:51 AM, N Keywal <nk...@gmail.com> wrote:
> > Hi Gary,
> >
> > The hbase specific versions for surefire & junit are ready (see
> HBASE-4763
> > for all the details).
> > Would it be possible to use your repository? What would you need from me?
> > I've never deployed anything in a maven repo. Surefire works with maven,
> so
> > it should be easy, JUnit does not, so there could be more work to do.
> >
> > I copy the dev list for information.
> >
> > Regards,
> >
> > N.
> >
> >
> > ---------- Forwarded message ----------
> > From: Gary Helmling <gh...@gmail.com>
> > Date: Thu, Oct 20, 2011 at 11:50 PM
> > Subject: Re: [unit tests] small tests vs. others
> > To: dev@hbase.apache.org
> >
> >
> > +1 on small/medium/large
> >
> > I have a test mvn repo up on people.apache.org that we can use for a
> > patched surefire build, if that helps.
> >
> >
> > On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray <jg...@fb.com> wrote:
> >> +1 on small/medium/large.
> >>
> >>> -----Original Message-----
> >>> From: Todd Lipcon [mailto:todd@cloudera.com]
> >>> Sent: Thursday, October 20, 2011 1:49 PM
> >>> To: dev@hbase.apache.org
> >>> Subject: Re: [unit tests] small tests vs. others
> >>>
> >>> +1 on separating into small/medium/large. One of the requirements of
> >>> "small" should be that it can run parallel with other tests and in
> nofork
> >>> mode.
> >>>
> >>> See SUREFIRE-329 for the necessary surefire patch. Until that's
> >>> integrated
> >>> upstream we'd have to have our own patched surefire :(
> >>>
> >>> -Todd
> >>>
> >>> On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
> >>> > Hi,
> >>> >
> >>> > I've done a stat on the tests we have. Many of them are actually
> quite
> >>> > simple: There are 226 test classes in HBase. 100 of them does not
> >>> > use a cluster, and takes less than 2 seconds to execute. 24 do not
> use
> >>> > a cluster but need between 2 and 40 seconds.
> >>> >
> >>> > I tried to execute these with different maven settings.
> >>> > fork: once or always, i.e. all the tests are executed in the same JVM
> >>> > or there is a new JVM created for every test class parallel;
> >>> > deactivated or activated, with 3 threads and a per class
> >>> > parallelisation.
> >>> >
> >>> > The 100 tests executes for the whole maven stuff, including the
> >>> > initial dependency & task analysis fork always: 1m50s fork once: 30
> >>> > seconds
> >>> > parallel: 30s (=> no difference).
> >>> >
> >>> > It works quite well, except with TestLruBlockCache that I had to
> >>> > remove.
> >>> >
> >>> > I tried as well with the 24 'bigger' ones but no cluster, for the
> >>> > following
> >>> > results:
> >>> > fork always: 7m36s
> >>> > parallel: 2m32s: 3 times faster.
> >>> >
> >>> > However, I had to remove many of them, there are some conflicts that
> I
> >>> > have not solved (nor tried to).
> >>> >
> >>> > The fact that nearly 50% of the tests can be executed in 1 minutes is
> >>> > interesting imho.
> >>> >
> >>> > That makes me wonder if we should try to get a separation of the
> tests
> >>> > in 3 parts, small/medium/large as of
> >>> >
> http://googletesting.blogspot.com/2009/07/software-testing-categorizat
> >>> > ion.html
> >>> >
> >>> > small would be:
> >>> > - less than two seconds, can be executed in the same jvm
> >>> > - all of them: less than 2 minute
> >>> > - no cluster, nothing that can break the environment
> >>> > - run by the developers as a part of the build
> >>> >
> >>> > medium:
> >>> > - less than 40 seconds
> >>> > - all of them: would last less than 30 minutes, ideally 15 minutes
> >>> > - run in an isolated jvm
> >>> > - run explicitly by the developers before submitting a fix
> >>> >
> >>> > large:
> >>> > - run in an isolated jvm
> >>> > - Should last less than 45minutes
> >>> > - run on the central built or explicitly by the developers
> >>> > - break the built when it fails.
> >>> >
> >>> > Only one issue: it's not directly possible with maven, but there are
> >>> > known workarounds :-)
> >>> >
> >>> >
> >>> > What do you think?
> >>> >
> >>> >
> >>> > Below the tests set for the 120 tests that seems to work altogether.
> >>> >
> >>> >
> >>> > mvn test
> >>> > -
> >>> Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
> >>> >
> se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
> >>> >
> ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
> >>> >
> TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
> >>> >
> >>> base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
> >>> >
> lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
> >>> >
> tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
> >>> > g.apache.hadoop.hbase.filter.TestFilterList,
> org.apache.hadoop.hbase.fi
> >>> >
> lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
> >>> >
> lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
> >>> >
> hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
> >>> >
> >>> mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
> >>> >
> Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
> >>> >
> apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
> >>> >
> e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
> >>> >
> >>> heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
> >>> k
> >>> >
> Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
> >>> >
> .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
> >>> >
> ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
> >>> >
> >>> pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
> >>> >
> ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
> >>> >
> >>> apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
> >>> e
> >>> >
> >>> .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable
> >>> >
> >>> ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
> >>> >
> >>> .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
> >>> .
> >>> >
> >>> master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
> >>> dS
> >>> >
> >>> erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach
> >>> >
> >>> e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
> >>> >
> >>> metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
> >>> Me
> >>> >
> >>> moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
> >>> Tas
> >>> >
> >>> kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
> >>> >
> >>> andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
> >>> >
> pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
> >>> >
> >>> adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
> >>> >
> >>> nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
> >>> V
> >>> >
> >>> alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
> >>> >
> >>> istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
> >>> e
> >>> >
> >>> .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
> >>> >
> >>> ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
> >>> >
> gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
> >>> >
> ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
> >>> >
> >>> he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h
> >>> >
> >>> base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas
> >>> >
> >>> e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse
> >>> >
> >>> rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
> >>> >
> >>> gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
> >>> p
> >>> >
> >>> ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
> >>> >
> >>> adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
> >>> e
> >>> >
> >>> l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
> >>> a
> >>> >
> >>> Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
> >>> doo
> >>> >
> >>> p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
> >>> el
> >>> >
> >>> .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test
> >>> >
> >>> StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl
> >>> >
> >>> eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
> >>> >
> >>> pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
> >>> .h
> >>> >
> >>> base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
> >>> de
> >>> >
> l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
> >>> >
> >>> e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
> >>> e,
> >>> >
> >>> org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
> >>> e
> >>> >
> >>> stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
> >>> >
> doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
> >>> >
> >>> apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
> >>> a
> >>> >
> se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
> >>> >
> p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
> >>> >
> t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
> >>> >
> p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
> >>> >
> >>> IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or
> >>> >
> >>> g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
> >>> >
> >>> stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
> >>> P
> >>> >
> >>> eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
> >>> .a
> >>> >
> >>> pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
> >>> >
> oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
> >>> >
> TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
> >>> >
> pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
> >>> >
> doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
> >>> >
> o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
> >>> >
> ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
> >>> >
> oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
> >>> >
> >>> le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache
> >>> >
> >>> .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
> >>> >
> >>> server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
> >>> e
> >>> >
> >>> mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
> >>> o
> >>> >
> ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
> >>> >
> >>> pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
> >>> >
> .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
> >>> > ReplicationSource
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >
> >
>
Re: [unit tests] small tests vs. others
Posted by Gary Helmling <gh...@gmail.com>.
Hi N,
I can push up the modified surefire and junit build with appropriate versions.
Is there anything special to either build that would complicate
building with the supplied patches? If not, then I can do the builds
and push up the artifacts. Then we coordinate the testing. How does
that sound?
--gh
On Mon, Nov 14, 2011 at 5:51 AM, N Keywal <nk...@gmail.com> wrote:
> Hi Gary,
>
> The hbase specific versions for surefire & junit are ready (see HBASE-4763
> for all the details).
> Would it be possible to use your repository? What would you need from me?
> I've never deployed anything in a maven repo. Surefire works with maven, so
> it should be easy, JUnit does not, so there could be more work to do.
>
> I copy the dev list for information.
>
> Regards,
>
> N.
>
>
> ---------- Forwarded message ----------
> From: Gary Helmling <gh...@gmail.com>
> Date: Thu, Oct 20, 2011 at 11:50 PM
> Subject: Re: [unit tests] small tests vs. others
> To: dev@hbase.apache.org
>
>
> +1 on small/medium/large
>
> I have a test mvn repo up on people.apache.org that we can use for a
> patched surefire build, if that helps.
>
>
> On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray <jg...@fb.com> wrote:
>> +1 on small/medium/large.
>>
>>> -----Original Message-----
>>> From: Todd Lipcon [mailto:todd@cloudera.com]
>>> Sent: Thursday, October 20, 2011 1:49 PM
>>> To: dev@hbase.apache.org
>>> Subject: Re: [unit tests] small tests vs. others
>>>
>>> +1 on separating into small/medium/large. One of the requirements of
>>> "small" should be that it can run parallel with other tests and in nofork
>>> mode.
>>>
>>> See SUREFIRE-329 for the necessary surefire patch. Until that's
>>> integrated
>>> upstream we'd have to have our own patched surefire :(
>>>
>>> -Todd
>>>
>>> On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I've done a stat on the tests we have. Many of them are actually quite
>>> > simple: There are 226 test classes in HBase. 100 of them does not
>>> > use a cluster, and takes less than 2 seconds to execute. 24 do not use
>>> > a cluster but need between 2 and 40 seconds.
>>> >
>>> > I tried to execute these with different maven settings.
>>> > fork: once or always, i.e. all the tests are executed in the same JVM
>>> > or there is a new JVM created for every test class parallel;
>>> > deactivated or activated, with 3 threads and a per class
>>> > parallelisation.
>>> >
>>> > The 100 tests executes for the whole maven stuff, including the
>>> > initial dependency & task analysis fork always: 1m50s fork once: 30
>>> > seconds
>>> > parallel: 30s (=> no difference).
>>> >
>>> > It works quite well, except with TestLruBlockCache that I had to
>>> > remove.
>>> >
>>> > I tried as well with the 24 'bigger' ones but no cluster, for the
>>> > following
>>> > results:
>>> > fork always: 7m36s
>>> > parallel: 2m32s: 3 times faster.
>>> >
>>> > However, I had to remove many of them, there are some conflicts that I
>>> > have not solved (nor tried to).
>>> >
>>> > The fact that nearly 50% of the tests can be executed in 1 minutes is
>>> > interesting imho.
>>> >
>>> > That makes me wonder if we should try to get a separation of the tests
>>> > in 3 parts, small/medium/large as of
>>> > http://googletesting.blogspot.com/2009/07/software-testing-categorizat
>>> > ion.html
>>> >
>>> > small would be:
>>> > - less than two seconds, can be executed in the same jvm
>>> > - all of them: less than 2 minute
>>> > - no cluster, nothing that can break the environment
>>> > - run by the developers as a part of the build
>>> >
>>> > medium:
>>> > - less than 40 seconds
>>> > - all of them: would last less than 30 minutes, ideally 15 minutes
>>> > - run in an isolated jvm
>>> > - run explicitly by the developers before submitting a fix
>>> >
>>> > large:
>>> > - run in an isolated jvm
>>> > - Should last less than 45minutes
>>> > - run on the central built or explicitly by the developers
>>> > - break the built when it fails.
>>> >
>>> > Only one issue: it's not directly possible with maven, but there are
>>> > known workarounds :-)
>>> >
>>> >
>>> > What do you think?
>>> >
>>> >
>>> > Below the tests set for the 120 tests that seems to work altogether.
>>> >
>>> >
>>> > mvn test
>>> > -
>>> Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
>>> > se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
>>> > ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
>>> > TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
>>> >
>>> base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
>>> > lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
>>> > tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
>>> > g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
>>> > lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
>>> > lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
>>> > hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
>>> >
>>> mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
>>> > Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
>>> > apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
>>> > e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
>>> >
>>> heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
>>> k
>>> > Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
>>> > .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
>>> > ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
>>> >
>>> pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
>>> > ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
>>> >
>>> apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
>>> e
>>> >
>>> .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable
>>> >
>>> ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
>>> >
>>> .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
>>> .
>>> >
>>> master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
>>> dS
>>> >
>>> erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach
>>> >
>>> e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
>>> >
>>> metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
>>> Me
>>> >
>>> moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
>>> Tas
>>> >
>>> kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
>>> >
>>> andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
>>> > pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
>>> >
>>> adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
>>> >
>>> nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
>>> V
>>> >
>>> alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
>>> >
>>> istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
>>> e
>>> >
>>> .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
>>> >
>>> ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
>>> > gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
>>> > ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
>>> >
>>> he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h
>>> >
>>> base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas
>>> >
>>> e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse
>>> >
>>> rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
>>> >
>>> gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
>>> p
>>> >
>>> ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
>>> >
>>> adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
>>> e
>>> >
>>> l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
>>> a
>>> >
>>> Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
>>> doo
>>> >
>>> p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
>>> el
>>> >
>>> .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test
>>> >
>>> StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl
>>> >
>>> eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
>>> >
>>> pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
>>> .h
>>> >
>>> base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
>>> de
>>> > l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
>>> >
>>> e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
>>> e,
>>> >
>>> org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
>>> e
>>> >
>>> stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
>>> > doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
>>> >
>>> apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
>>> a
>>> > se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
>>> > p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
>>> > t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
>>> > p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
>>> >
>>> IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or
>>> >
>>> g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
>>> >
>>> stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
>>> P
>>> >
>>> eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
>>> .a
>>> >
>>> pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
>>> > oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
>>> > TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
>>> > pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
>>> > doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
>>> > o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
>>> > ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
>>> > oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
>>> >
>>> le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache
>>> >
>>> .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
>>> >
>>> server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
>>> e
>>> >
>>> mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
>>> o
>>> > ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
>>> >
>>> pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
>>> > .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
>>> > ReplicationSource
>>> >
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>
>
Fwd: [unit tests] small tests vs. others
Posted by N Keywal <nk...@gmail.com>.
Hi Gary,
The hbase specific versions for surefire & junit are ready (see HBASE-4763
for all the details).
Would it be possible to use your repository? What would you need from me?
I've never deployed anything in a maven repo. Surefire works with maven, so
it should be easy, JUnit does not, so there could be more work to do.
I copy the dev list for information.
Regards,
N.
---------- Forwarded message ----------
From: Gary Helmling <gh...@gmail.com>
Date: Thu, Oct 20, 2011 at 11:50 PM
Subject: Re: [unit tests] small tests vs. others
To: dev@hbase.apache.org
+1 on small/medium/large
I have a test mvn repo up on people.apache.org that we can use for a
patched surefire build, if that helps.
On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray <jg...@fb.com> wrote:
> +1 on small/medium/large.
>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, October 20, 2011 1:49 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [unit tests] small tests vs. others
>>
>> +1 on separating into small/medium/large. One of the requirements of
>> "small" should be that it can run parallel with other tests and in
nofork mode.
>>
>> See SUREFIRE-329 for the necessary surefire patch. Until that's
integrated
>> upstream we'd have to have our own patched surefire :(
>>
>> -Todd
>>
>> On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
>> > Hi,
>> >
>> > I've done a stat on the tests we have. Many of them are actually quite
>> > simple: There are 226 test classes in HBase. 100 of them does not
>> > use a cluster, and takes less than 2 seconds to execute. 24 do not use
>> > a cluster but need between 2 and 40 seconds.
>> >
>> > I tried to execute these with different maven settings.
>> > fork: once or always, i.e. all the tests are executed in the same JVM
>> > or there is a new JVM created for every test class parallel;
>> > deactivated or activated, with 3 threads and a per class
>> > parallelisation.
>> >
>> > The 100 tests executes for the whole maven stuff, including the
>> > initial dependency & task analysis fork always: 1m50s fork once: 30
>> > seconds
>> > parallel: 30s (=> no difference).
>> >
>> > It works quite well, except with TestLruBlockCache that I had to
remove.
>> >
>> > I tried as well with the 24 'bigger' ones but no cluster, for the
>> > following
>> > results:
>> > fork always: 7m36s
>> > parallel: 2m32s: 3 times faster.
>> >
>> > However, I had to remove many of them, there are some conflicts that I
>> > have not solved (nor tried to).
>> >
>> > The fact that nearly 50% of the tests can be executed in 1 minutes is
>> > interesting imho.
>> >
>> > That makes me wonder if we should try to get a separation of the tests
>> > in 3 parts, small/medium/large as of
>> > http://googletesting.blogspot.com/2009/07/software-testing-categorizat
>> > ion.html
>> >
>> > small would be:
>> > - less than two seconds, can be executed in the same jvm
>> > - all of them: less than 2 minute
>> > - no cluster, nothing that can break the environment
>> > - run by the developers as a part of the build
>> >
>> > medium:
>> > - less than 40 seconds
>> > - all of them: would last less than 30 minutes, ideally 15 minutes
>> > - run in an isolated jvm
>> > - run explicitly by the developers before submitting a fix
>> >
>> > large:
>> > - run in an isolated jvm
>> > - Should last less than 45minutes
>> > - run on the central built or explicitly by the developers
>> > - break the built when it fails.
>> >
>> > Only one issue: it's not directly possible with maven, but there are
>> > known workarounds :-)
>> >
>> >
>> > What do you think?
>> >
>> >
>> > Below the tests set for the 120 tests that seems to work altogether.
>> >
>> >
>> > mvn test
>> > -
>> Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
>> > se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
>> > ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
>> > TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
>> >
>> base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
>> > lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
>> > tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
>> > g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
>> > lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
>> > lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
>> > hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
>> >
>> mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
>> > Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
>> > apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
>> > e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
>> >
>> heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
>> k
>> > Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
>> > .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
>> > ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
>> >
>> pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
>> > ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
>> >
>> apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
>> e
>> >
>> .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable
>> >
>> ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
>> >
>> .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
>> .
>> >
>> master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
>> dS
>> >
>> erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach
>> >
>> e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
>> >
>> metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
>> Me
>> >
>> moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
>> Tas
>> >
>> kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
>> >
>> andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
>> > pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
>> >
>> adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
>> >
>> nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
>> V
>> >
>> alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
>> >
>> istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
>> e
>> >
>> .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
>> >
>> ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
>> > gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
>> > ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
>> >
>> he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h
>> >
>> base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas
>> >
>> e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse
>> >
>> rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
>> >
>> gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
>> p
>> >
>> ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
>> >
>> adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
>> e
>> >
>> l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
>> a
>> >
>> Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
>> doo
>> >
>> p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
>> el
>> >
>> .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test
>> >
>> StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl
>> >
>> eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
>> >
>> pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
>> .h
>> >
>> base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
>> de
>> > l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
>> >
>> e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
>> e,
>> >
>> org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
>> e
>> >
>> stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
>> > doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
>> >
>> apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
>> a
>> > se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
>> > p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
>> > t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
>> > p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
>> >
>> IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or
>> >
>> g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
>> >
>> stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
>> P
>> >
>> eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
>> .a
>> >
>> pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
>> > oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
>> > TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
>> > pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
>> > doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
>> > o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
>> > ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
>> > oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
>> >
>> le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache
>> >
>> .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
>> >
>> server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
>> e
>> >
>> mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
>> o
>> > ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
>> >
>> pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
>> > .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
>> > ReplicationSource
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
Re: [unit tests] small tests vs. others
Posted by Gary Helmling <gh...@gmail.com>.
+1 on small/medium/large
I have a test mvn repo up on people.apache.org that we can use for a
patched surefire build, if that helps.
On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray <jg...@fb.com> wrote:
> +1 on small/medium/large.
>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, October 20, 2011 1:49 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [unit tests] small tests vs. others
>>
>> +1 on separating into small/medium/large. One of the requirements of
>> "small" should be that it can run parallel with other tests and in nofork mode.
>>
>> See SUREFIRE-329 for the necessary surefire patch. Until that's integrated
>> upstream we'd have to have our own patched surefire :(
>>
>> -Todd
>>
>> On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
>> > Hi,
>> >
>> > I've done a stat on the tests we have. Many of them are actually quite
>> > simple: There are 226 test classes in HBase. 100 of them does not
>> > use a cluster, and takes less than 2 seconds to execute. 24 do not use
>> > a cluster but need between 2 and 40 seconds.
>> >
>> > I tried to execute these with different maven settings.
>> > fork: once or always, i.e. all the tests are executed in the same JVM
>> > or there is a new JVM created for every test class parallel;
>> > deactivated or activated, with 3 threads and a per class
>> > parallelisation.
>> >
>> > The 100 tests executes for the whole maven stuff, including the
>> > initial dependency & task analysis fork always: 1m50s fork once: 30
>> > seconds
>> > parallel: 30s (=> no difference).
>> >
>> > It works quite well, except with TestLruBlockCache that I had to remove.
>> >
>> > I tried as well with the 24 'bigger' ones but no cluster, for the
>> > following
>> > results:
>> > fork always: 7m36s
>> > parallel: 2m32s: 3 times faster.
>> >
>> > However, I had to remove many of them, there are some conflicts that I
>> > have not solved (nor tried to).
>> >
>> > The fact that nearly 50% of the tests can be executed in 1 minutes is
>> > interesting imho.
>> >
>> > That makes me wonder if we should try to get a separation of the tests
>> > in 3 parts, small/medium/large as of
>> > http://googletesting.blogspot.com/2009/07/software-testing-categorizat
>> > ion.html
>> >
>> > small would be:
>> > - less than two seconds, can be executed in the same jvm
>> > - all of them: less than 2 minute
>> > - no cluster, nothing that can break the environment
>> > - run by the developers as a part of the build
>> >
>> > medium:
>> > - less than 40 seconds
>> > - all of them: would last less than 30 minutes, ideally 15 minutes
>> > - run in an isolated jvm
>> > - run explicitly by the developers before submitting a fix
>> >
>> > large:
>> > - run in an isolated jvm
>> > - Should last less than 45minutes
>> > - run on the central built or explicitly by the developers
>> > - break the built when it fails.
>> >
>> > Only one issue: it's not directly possible with maven, but there are
>> > known workarounds :-)
>> >
>> >
>> > What do you think?
>> >
>> >
>> > Below the tests set for the 120 tests that seems to work altogether.
>> >
>> >
>> > mvn test
>> > -
>> Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
>> > se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
>> > ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
>> > TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
>> >
>> base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
>> > lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
>> > tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
>> > g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
>> > lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
>> > lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
>> > hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
>> >
>> mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
>> > Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
>> > apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
>> > e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
>> >
>> heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
>> k
>> > Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
>> > .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
>> > ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
>> >
>> pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
>> > ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
>> >
>> apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
>> e
>> >
>> .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable
>> >
>> ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
>> >
>> .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
>> .
>> >
>> master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
>> dS
>> >
>> erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach
>> >
>> e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
>> >
>> metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
>> Me
>> >
>> moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
>> Tas
>> >
>> kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
>> >
>> andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
>> > pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
>> >
>> adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
>> >
>> nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
>> V
>> >
>> alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
>> >
>> istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
>> e
>> >
>> .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
>> >
>> ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
>> > gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
>> > ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
>> >
>> he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h
>> >
>> base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas
>> >
>> e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse
>> >
>> rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
>> >
>> gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
>> p
>> >
>> ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
>> >
>> adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
>> e
>> >
>> l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
>> a
>> >
>> Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
>> doo
>> >
>> p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
>> el
>> >
>> .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test
>> >
>> StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl
>> >
>> eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
>> >
>> pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
>> .h
>> >
>> base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
>> de
>> > l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
>> >
>> e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
>> e,
>> >
>> org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
>> e
>> >
>> stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
>> > doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
>> >
>> apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
>> a
>> > se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
>> > p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
>> > t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
>> > p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
>> >
>> IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or
>> >
>> g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
>> >
>> stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
>> P
>> >
>> eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
>> .a
>> >
>> pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
>> > oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
>> > TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
>> > pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
>> > doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
>> > o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
>> > ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
>> > oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
>> >
>> le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache
>> >
>> .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
>> >
>> server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
>> e
>> >
>> mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
>> o
>> > ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
>> >
>> pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
>> > .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
>> > ReplicationSource
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
RE: [unit tests] small tests vs. others
Posted by Jonathan Gray <jg...@fb.com>.
+1 on small/medium/large.
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, October 20, 2011 1:49 PM
> To: dev@hbase.apache.org
> Subject: Re: [unit tests] small tests vs. others
>
> +1 on separating into small/medium/large. One of the requirements of
> "small" should be that it can run parallel with other tests and in nofork mode.
>
> See SUREFIRE-329 for the necessary surefire patch. Until that's integrated
> upstream we'd have to have our own patched surefire :(
>
> -Todd
>
> On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
> > Hi,
> >
> > I've done a stat on the tests we have. Many of them are actually quite
> > simple: There are 226 test classes in HBase. 100 of them does not
> > use a cluster, and takes less than 2 seconds to execute. 24 do not use
> > a cluster but need between 2 and 40 seconds.
> >
> > I tried to execute these with different maven settings.
> > fork: once or always, i.e. all the tests are executed in the same JVM
> > or there is a new JVM created for every test class parallel;
> > deactivated or activated, with 3 threads and a per class
> > parallelisation.
> >
> > The 100 tests executes for the whole maven stuff, including the
> > initial dependency & task analysis fork always: 1m50s fork once: 30
> > seconds
> > parallel: 30s (=> no difference).
> >
> > It works quite well, except with TestLruBlockCache that I had to remove.
> >
> > I tried as well with the 24 'bigger' ones but no cluster, for the
> > following
> > results:
> > fork always: 7m36s
> > parallel: 2m32s: 3 times faster.
> >
> > However, I had to remove many of them, there are some conflicts that I
> > have not solved (nor tried to).
> >
> > The fact that nearly 50% of the tests can be executed in 1 minutes is
> > interesting imho.
> >
> > That makes me wonder if we should try to get a separation of the tests
> > in 3 parts, small/medium/large as of
> > http://googletesting.blogspot.com/2009/07/software-testing-categorizat
> > ion.html
> >
> > small would be:
> > - less than two seconds, can be executed in the same jvm
> > - all of them: less than 2 minute
> > - no cluster, nothing that can break the environment
> > - run by the developers as a part of the build
> >
> > medium:
> > - less than 40 seconds
> > - all of them: would last less than 30 minutes, ideally 15 minutes
> > - run in an isolated jvm
> > - run explicitly by the developers before submitting a fix
> >
> > large:
> > - run in an isolated jvm
> > - Should last less than 45minutes
> > - run on the central built or explicitly by the developers
> > - break the built when it fails.
> >
> > Only one issue: it's not directly possible with maven, but there are
> > known workarounds :-)
> >
> >
> > What do you think?
> >
> >
> > Below the tests set for the 120 tests that seems to work altogether.
> >
> >
> > mvn test
> > -
> Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
> > se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
> > ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
> > TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
> >
> base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
> > lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
> > tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
> > g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
> > lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
> > lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
> > hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
> >
> mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
> > Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
> > apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
> > e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
> >
> heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
> k
> > Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
> > .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
> > ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
> >
> pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
> > ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
> >
> apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
> e
> >
> .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable
> >
> ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
> >
> .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
> .
> >
> master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
> dS
> >
> erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach
> >
> e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
> >
> metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
> Me
> >
> moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
> Tas
> >
> kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
> >
> andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
> > pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
> >
> adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
> >
> nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
> V
> >
> alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
> >
> istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
> e
> >
> .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
> >
> ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
> > gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
> > ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
> >
> he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h
> >
> base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas
> >
> e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse
> >
> rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
> >
> gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
> p
> >
> ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
> >
> adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
> e
> >
> l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
> a
> >
> Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
> doo
> >
> p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
> el
> >
> .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test
> >
> StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl
> >
> eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
> >
> pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
> .h
> >
> base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
> de
> > l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
> >
> e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
> e,
> >
> org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
> e
> >
> stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
> > doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
> >
> apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
> a
> > se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
> > p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
> > t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
> > p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
> >
> IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or
> >
> g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
> >
> stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
> P
> >
> eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
> .a
> >
> pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
> > oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
> > TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
> > pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
> > doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
> > o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
> > ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
> > oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
> >
> le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache
> >
> .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
> >
> server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
> e
> >
> mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
> o
> > ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
> >
> pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
> > .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
> > ReplicationSource
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: [unit tests] small tests vs. others
Posted by Todd Lipcon <to...@cloudera.com>.
+1 on separating into small/medium/large. One of the requirements of
"small" should be that it can run parallel with other tests and in
nofork mode.
See SUREFIRE-329 for the necessary surefire patch. Until that's
integrated upstream we'd have to have our own patched surefire :(
-Todd
On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
> Hi,
>
> I've done a stat on the tests we have. Many of them are actually quite
> simple: There are 226 test classes in HBase. 100 of them does not use a
> cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
> but need between 2 and 40 seconds.
>
> I tried to execute these with different maven settings.
> fork: once or always, i.e. all the tests are executed in the same JVM or
> there is a new JVM created for every test class
> parallel; deactivated or activated, with 3 threads and a per class
> parallelisation.
>
> The 100 tests executes for the whole maven stuff, including the initial
> dependency & task analysis
> fork always: 1m50s
> fork once: 30 seconds
> parallel: 30s (=> no difference).
>
> It works quite well, except with TestLruBlockCache that I had to remove.
>
> I tried as well with the 24 'bigger' ones but no cluster, for the following
> results:
> fork always: 7m36s
> parallel: 2m32s: 3 times faster.
>
> However, I had to remove many of them, there are some conflicts that I have
> not solved (nor tried to).
>
> The fact that nearly 50% of the tests can be executed in 1 minutes is
> interesting imho.
>
> That makes me wonder if we should try to get a separation of the tests in 3
> parts, small/medium/large as of
> http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
>
> small would be:
> - less than two seconds, can be executed in the same jvm
> - all of them: less than 2 minute
> - no cluster, nothing that can break the environment
> - run by the developers as a part of the build
>
> medium:
> - less than 40 seconds
> - all of them: would last less than 30 minutes, ideally 15 minutes
> - run in an isolated jvm
> - run explicitly by the developers before submitting a fix
>
> large:
> - run in an isolated jvm
> - Should last less than 45minutes
> - run on the central built or explicitly by the developers
> - break the built when it fails.
>
> Only one issue: it's not directly possible with maven, but there are known
> workarounds :-)
>
>
> What do you think?
>
>
> Below the tests set for the 120 tests that seems to work altogether.
>
>
> mvn test
> -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: [unit tests] small tests vs. others
Posted by Akash Ashok <th...@gmail.com>.
Ooo +1 on this. This sounds Just wonderful. :)
Cheers,
Akash A
On Fri, Oct 21, 2011 at 3:34 AM, Andrew Purtell <ap...@apache.org> wrote:
> +1, this sounds great.
>
> In 0.20 I had the REST (aka Stargate at that time) tests running all in
> single JVM but in the port to 0.90 had to be updated for the current
> surefire config. If there were a small/medium convention at that time the
> relative speed could have been preserved.
>
> - Andy
>
>
>
>
>
> >________________________________
> >From: N Keywal <nk...@gmail.com>
> >To: dev@hbase.apache.org
> >Sent: Thursday, October 20, 2011 1:46 PM
> >Subject: [unit tests] small tests vs. others
> >
> >Hi,
> >
> >I've done a stat on the tests we have. Many of them are actually quite
> >simple: There are 226 test classes in HBase. 100 of them does not use a
> >cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
> >but need between 2 and 40 seconds.
> >
> >I tried to execute these with different maven settings.
> >fork: once or always, i.e. all the tests are executed in the same JVM or
> >there is a new JVM created for every test class
> >parallel; deactivated or activated, with 3 threads and a per class
> >parallelisation.
> >
> >The 100 tests executes for the whole maven stuff, including the initial
> >dependency & task analysis
> >fork always: 1m50s
> >fork once: 30 seconds
> >parallel: 30s (=> no difference).
> >
> >It works quite well, except with TestLruBlockCache that I had to remove.
> >
> >I tried as well with the 24 'bigger' ones but no cluster, for the
> following
> >results:
> >fork always: 7m36s
> >parallel: 2m32s: 3 times faster.
> >
> >However, I had to remove many of them, there are some conflicts that I
> have
> >not solved (nor tried to).
> >
> >The fact that nearly 50% of the tests can be executed in 1 minutes is
> >interesting imho.
> >
> >That makes me wonder if we should try to get a separation of the tests in
> 3
> >parts, small/medium/large as of
> >
> http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
> >
> >small would be:
> >- less than two seconds, can be executed in the same jvm
> >- all of them: less than 2 minute
> >- no cluster, nothing that can break the environment
> >- run by the developers as a part of the build
> >
> >medium:
> >- less than 40 seconds
> >- all of them: would last less than 30 minutes, ideally 15 minutes
> >- run in an isolated jvm
> >- run explicitly by the developers before submitting a fix
> >
> >large:
> >- run in an isolated jvm
> >- Should last less than 45minutes
> >- run on the central built or explicitly by the developers
> >- break the built when it fails.
> >
> >Only one issue: it's not directly possible with maven, but there are known
> >workarounds :-)
> >
> >
> >What do you think?
> >
> >
> >Below the tests set for the 120 tests that seems to work altogether.
> >
> >
> >mvn test
>
> >-Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apac
>
> he.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServl
>
> et,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.T
>
> estScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.T
>
> estTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,
>
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizati
>
> ons,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
> >
> >
> >
>
Re: [unit tests] small tests vs. others
Posted by Andrew Purtell <ap...@apache.org>.
+1, this sounds great.
In 0.20 I had the REST (aka Stargate at that time) tests running all in single JVM but in the port to 0.90 had to be updated for the current surefire config. If there were a small/medium convention at that time the relative speed could have been preserved.
- Andy
>________________________________
>From: N Keywal <nk...@gmail.com>
>To: dev@hbase.apache.org
>Sent: Thursday, October 20, 2011 1:46 PM
>Subject: [unit tests] small tests vs. others
>
>Hi,
>
>I've done a stat on the tests we have. Many of them are actually quite
>simple: There are 226 test classes in HBase. 100 of them does not use a
>cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
>but need between 2 and 40 seconds.
>
>I tried to execute these with different maven settings.
>fork: once or always, i.e. all the tests are executed in the same JVM or
>there is a new JVM created for every test class
>parallel; deactivated or activated, with 3 threads and a per class
>parallelisation.
>
>The 100 tests executes for the whole maven stuff, including the initial
>dependency & task analysis
>fork always: 1m50s
>fork once: 30 seconds
>parallel: 30s (=> no difference).
>
>It works quite well, except with TestLruBlockCache that I had to remove.
>
>I tried as well with the 24 'bigger' ones but no cluster, for the following
>results:
>fork always: 7m36s
>parallel: 2m32s: 3 times faster.
>
>However, I had to remove many of them, there are some conflicts that I have
>not solved (nor tried to).
>
>The fact that nearly 50% of the tests can be executed in 1 minutes is
>interesting imho.
>
>That makes me wonder if we should try to get a separation of the tests in 3
>parts, small/medium/large as of
>http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
>
>small would be:
>- less than two seconds, can be executed in the same jvm
>- all of them: less than 2 minute
>- no cluster, nothing that can break the environment
>- run by the developers as a part of the build
>
>medium:
>- less than 40 seconds
>- all of them: would last less than 30 minutes, ideally 15 minutes
>- run in an isolated jvm
>- run explicitly by the developers before submitting a fix
>
>large:
>- run in an isolated jvm
>- Should last less than 45minutes
>- run on the central built or explicitly by the developers
>- break the built when it fails.
>
>Only one issue: it's not directly possible with maven, but there are known
>workarounds :-)
>
>
>What do you think?
>
>
>Below the tests set for the 120 tests that seems to work altogether.
>
>
>mvn test
>-Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apac
he.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServl
et,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.T
estScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.T
estTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,
org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizati
ons,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
>
>
>
Re: [unit tests] small tests vs. others
Posted by Stack <st...@duboce.net>.
Sounds good Nicolas.
St.Ack
On Thu, Oct 20, 2011 at 1:46 PM, N Keywal <nk...@gmail.com> wrote:
> Hi,
>
> I've done a stat on the tests we have. Many of them are actually quite
> simple: There are 226 test classes in HBase. 100 of them does not use a
> cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
> but need between 2 and 40 seconds.
>
> I tried to execute these with different maven settings.
> fork: once or always, i.e. all the tests are executed in the same JVM or
> there is a new JVM created for every test class
> parallel; deactivated or activated, with 3 threads and a per class
> parallelisation.
>
> The 100 tests executes for the whole maven stuff, including the initial
> dependency & task analysis
> fork always: 1m50s
> fork once: 30 seconds
> parallel: 30s (=> no difference).
>
> It works quite well, except with TestLruBlockCache that I had to remove.
>
> I tried as well with the 24 'bigger' ones but no cluster, for the following
> results:
> fork always: 7m36s
> parallel: 2m32s: 3 times faster.
>
> However, I had to remove many of them, there are some conflicts that I have
> not solved (nor tried to).
>
> The fact that nearly 50% of the tests can be executed in 1 minutes is
> interesting imho.
>
> That makes me wonder if we should try to get a separation of the tests in 3
> parts, small/medium/large as of
> http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
>
> small would be:
> - less than two seconds, can be executed in the same jvm
> - all of them: less than 2 minute
> - no cluster, nothing that can break the environment
> - run by the developers as a part of the build
>
> medium:
> - less than 40 seconds
> - all of them: would last less than 30 minutes, ideally 15 minutes
> - run in an isolated jvm
> - run explicitly by the developers before submitting a fix
>
> large:
> - run in an isolated jvm
> - Should last less than 45minutes
> - run on the central built or explicitly by the developers
> - break the built when it fails.
>
> Only one issue: it's not directly possible with maven, but there are known
> workarounds :-)
>
>
> What do you think?
>
>
> Below the tests set for the 120 tests that seems to work altogether.
>
>
> mvn test
> -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
>