You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2011/11/03 16:19:32 UTC
[jira] [Created] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
--------------------------------------------------------------------------------------------------
Key: HBASE-4737
URL: https://issues.apache.org/jira/browse/HBASE-4737
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
1) Split the tests in 3 categories
- small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
- medium: <45s, no flaky, useful to detect bugs immediatly
- large: remaining
2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
- will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
Proposed split:
Small:122 classes, 479 methods, ~3 minutes when no //)
Medium: 78 classes, 373 methods, ~23 minutes
Large: 34 classes, 221 methods, ~60 minutes
I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
MEDIUM LIST (name; number of methods, time)
org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
org.apache.hadoop.hbase.client.TestHCM 3 21.961
org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
org.apache.hadoop.hbase.master.TestMaster 1 18.918
org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
org.apache.hadoop.hbase.rest.TestTransform 1 15.966
org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
org.apache.hadoop.hbase.TestInfoServers 2 14.9
org.apache.hadoop.hbase.TestMultiVersions 3 27.945
org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
LARGE
org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
org.apache.hadoop.hbase.client.TestAdmin 33 369.833
org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
org.apache.hadoop.hbase.client.TestShell 1 69.65
org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
org.apache.hadoop.hbase.replication.TestReplication 7 163.158
org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
org.apache.hadoop.hbase.TestZooKeeper 7 86.267
org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
SMALL
Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 4737_part2.v1.patch
tag the small tests. retags some tests.
Does not modify executed code.
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.v2.patch
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 4737_all.patch
impacts on java files. Just JUnit categories.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146689#comment-13146689 ]
nkeywal commented on HBASE-4737:
--------------------------------
@commiters: are you ok with the patch? The pom has been already commited. The important one is 4737_all.patch because it impacts many test files, so it will have conflict if we wait too much. It does not modify the executed code itself, it just adds categories. The shell script can wait if necessary.
Note that I will have to write another patch for the small tests, but I believe it's simpler if done in two steps vs. one.
thank you :-)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4737:
--------------------------
Description:
1) Split the tests in 3 categories
- small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
- medium: <45s, no flaky, useful to detect bugs immediatly
- large: remaining
2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
- will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
Proposed split:
Small:122 classes, 479 methods, ~3 minutes when no //)
Medium: 78 classes, 373 methods, ~23 minutes
Large: 34 classes, 221 methods, ~60 minutes
I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
MEDIUM LIST (name; number of methods, time)
org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
org.apache.hadoop.hbase.client.TestHCM 3 21.961
org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
org.apache.hadoop.hbase.master.TestMaster 1 18.918
org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
org.apache.hadoop.hbase.rest.TestTransform 1 15.966
org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
org.apache.hadoop.hbase.TestInfoServers 2 14.9
org.apache.hadoop.hbase.TestMultiVersions 3 27.945
org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
LARGE
org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
org.apache.hadoop.hbase.client.TestAdmin 33 369.833
org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
org.apache.hadoop.hbase.client.TestShell 1 69.65
org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
org.apache.hadoop.hbase.replication.TestReplication 7 163.158
org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
org.apache.hadoop.hbase.TestZooKeeper 7 86.267
org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
SMALL
Everyting else
was:
1) Split the tests in 3 categories
- small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
- medium: <45s, no flaky, useful to detect bugs immediatly
- large: remaining
2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
- will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
Proposed split:
Small:122 classes, 479 methods, ~3 minutes when no //)
Medium: 78 classes, 373 methods, ~23 minutes
Large: 34 classes, 221 methods, ~60 minutes
I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
MEDIUM LIST (name; number of methods, time)
org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
org.apache.hadoop.hbase.client.TestHCM 3 21.961
org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
org.apache.hadoop.hbase.master.TestMaster 1 18.918
org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
org.apache.hadoop.hbase.rest.TestTransform 1 15.966
org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
org.apache.hadoop.hbase.TestInfoServers 2 14.9
org.apache.hadoop.hbase.TestMultiVersions 3 27.945
org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
LARGE
org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
org.apache.hadoop.hbase.client.TestAdmin 33 369.833
org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
org.apache.hadoop.hbase.client.TestShell 1 69.65
org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
org.apache.hadoop.hbase.replication.TestReplication 7 163.158
org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
org.apache.hadoop.hbase.TestZooKeeper 7 86.267
org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
SMALL
Everyting else
Fix Version/s: 0.94.0
Summary: Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM (was: Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM)
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.patch
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
one more time
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147341#comment-13147341 ]
Ted Yu commented on HBASE-4737:
-------------------------------
TestDistributedLogSplitting passed for me.
Patch for tagging SmallTests has been integrated.
Thanks N.
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147522#comment-13147522 ]
Hudson commented on HBASE-4737:
-------------------------------
Integrated in HBase-TRUNK #2425 (See [https://builds.apache.org/job/HBase-TRUNK/2425/])
HBASE-4737 Addendum removes unnecessary print
tedyu :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737.addendum, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144626#comment-13144626 ]
nkeywal commented on HBASE-4737:
--------------------------------
bq. The failed tests from other groups weren't included in the above list.
What do you mean?
Note that surefire is buggy, and the results presented while running may be wrong. For example, it can says that a class tests have 5 errors. When you look at the file it wrote, it says that the 2 tests are ok, and in the class there are actually two tests methods, not five. If you generate the report at the end with surefire-report it's fine however. I am going to mention this in the script.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144884#comment-13144884 ]
Mikhail Bautin commented on HBASE-4737:
---------------------------------------
I tried to run trunk unit tests using my map-reduce test runner (which unfortunately only works in Facebook environment at the moment). To my surprise, it worked extremely well without HBASE-4746, confirming N's observation. The job completed in under 10 minutes, just like the one for 89-fb, and ran 239 test classes with 1096 tests. The only failure was TestHRegionInfo.testGetSetOfHTD (an NPE; will investigate). I started working on HBASE-4767 patch for trunk, but most of its value will be in refactoring and cleanup, since it looks like there are very few bugs left in the trunk related to not passing configuration from MiniHBaseCluster to clients in unit tests.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143932#comment-13143932 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502384/20111103_4737_pom.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/170//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/170//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/170//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147416#comment-13147416 ]
Hudson commented on HBASE-4737:
-------------------------------
Integrated in HBase-TRUNK #2424 (See [https://builds.apache.org/job/HBase-TRUNK/2424/])
HBASE-4737 Addendum tags SmallTests
tedyu :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestCompare.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHRegionLocation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHServerInfo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestSerialization.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/avro/TestAvroUtil.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestResult.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestScan.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestBitComparator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestInclusiveStopFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestPageFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestPrefixFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestRandomRowFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueExcludeFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestImmutableBytesWritable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestSimpleTotalOrderPartitioner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSplit.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestClockSkewDetection.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/migration/TestMigrationFrom090To092.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/monitoring/TestMemoryBoundedLogMessageBuffer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueSkipListSet.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestReadWriteConsistencyControl.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanDeleteTracker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestCellSetModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestColumnSchemaModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestRowModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestScannerModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterStatusModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestStorageClusterVersionModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableInfoModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableListModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestTableSchemaModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/model/TestVersionModel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/security/TestUser.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestByteBloomFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestKeying.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestSortedCopyOnWriteSet.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperMainServerArg.java
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143255#comment-13143255 ]
nkeywal commented on HBASE-4737:
--------------------------------
I didn't know. But I believe there could be side effects on non necessary
before & after Class. For example, in io.hfile.TestHFileBlock#
testBlockHeapSize you don't need to start a cluster, so I expect it's
necessary to move it outside of the class if we want to put it in the small
'no cluster' category?
On Thu, Nov 3, 2011 at 4:47 PM, Todd Lipcon (Commented) (JIRA) <
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: MediumTests.java
SmallTests.java
LargeTests.java
4737_all.v2.patch
The diff --cached doesn't include the license, so I put the 3 files and the patch.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4737:
--------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143324#comment-13143324 ]
stack commented on HBASE-4737:
------------------------------
What happened? It ran tests for 15 minutes then gave up?
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144705#comment-13144705 ]
nkeywal commented on HBASE-4737:
--------------------------------
I've done test on the trunk and on the previous version (2 days old), it was working well, even if I knew there is some randomness (that I am going to tackle :-). On my last try, I had one error that I don't have without parallelization, in thrift2.TestThriftHBaseServiceHandler. It's zookeepeer related.
I am surprised by it works so well without HBASE-4746 actually. Do you know if the facebook team is going to push it on trunk soon? If not I can try to do it.
But in your case, for example client.TestGet is in the "small" list. There is no cluster/zookeeper for tests in this category, so it's something different, and it should work already.
Could you send me the content of you surefire-report directory? The final report is quite simple: if there is no className.txt file for the test, or if this file is empty, it's counted as not finished. This because surefire generates a report when the test is finished.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4737:
--------------------------
Attachment: 4737.addendum
Addendum removes the print
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737.addendum, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143288#comment-13143288 ]
stack commented on HBASE-4737:
------------------------------
I think you have to reupload the patch N.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146702#comment-13146702 ]
Ted Yu commented on HBASE-4737:
-------------------------------
@N:
Can you add Apache license to the new classes ?
Thanks
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4737:
--------------------------
Unit tests are categorized into three categories:
- small: no cluster, shorter than 15s, can be run in parallel with other tests in a JVM. Untagged.
- medium: shorter than 45s, not flaky, useful to detect bugs immediately. Tagged with @Category(MediumTests.class)
- large: remaining tests. Tagged with @Category(LargeTests.class)
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146713#comment-13146713 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503000/MediumTests.java
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/218//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147206#comment-13147206 ]
nkeywal commented on HBASE-4737:
--------------------------------
Hi,
I am fighting with surefire (and it's a difficult fight), but I will send
an update today...
On Wed, Nov 9, 2011 at 7:09 PM, Jesse Yates (Commented) (JIRA) <
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.dummy.patch
dummy patch to test the build. does nothing
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144349#comment-13144349 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Still on MacBook.
The results from shell script left something to be desired:
{code}
227 tests didn't finish
Tests in error are: mapreduce.TestHFileOutputFormat, master.TestDistributedLogSplitting
Tests that didn't finish are: .client.TestAttributes, .client.TestGet, .client.TestOperation, .client.TestResult, .client.TestScan, .coprocessor.TestCoprocessorInterface, .coprocessor.TestRegionObserverStacking, .executor.TestExecutorService, .filter.TestBitComparator, .filter.TestColumnPaginationFilter, .filter.TestColumnPrefixFilter, .filter.TestDependentColumnFilter, .filter.TestFilter, .filter.TestFilterList, .filter.TestInclusiveStopFilter, .filter.TestMultipleColumnPrefixFilter, .filter.TestPageFilter, .filter.TestParseFilter, .filter.TestPrefixFilter, .filter.TestRandomRowFilter, .filter.TestSingleColumnValueExcludeFilter, .filter.TestSingleColumnValueFilter,...
{code}
The failed tests from other groups weren't included in the above list.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144118#comment-13144118 ]
stack commented on HBASE-4737:
------------------------------
Hmm... maybe hbase-4553 last patch is something else...
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144130#comment-13144130 ]
nkeywal commented on HBASE-4737:
--------------------------------
The default profile is not taken into account, I don't know why. But putting the property in the project itself should work. That's what the v2 is about. With the v1, I think that the process was forked only once for all tests that broke hbase tests then surefire :-/
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145275#comment-13145275 ]
nkeywal commented on HBASE-4737:
--------------------------------
the -1 on @author is a bug in the validation tool: I actually removed an author tag, not added one.
Other errors are the usual ones (this patch does not modify the behavior; just adds tags). So I think it's ok for integration. Parallel builds can be tested with the shell script.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146704#comment-13146704 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Can we realign the categories (especially large tests) according to real execution time reported by Jenkins ?
e.g.
{code}
+@Category(LargeTests.class)
public class TestRegionRebalancing {
{code}
According to https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2422/testReport/org.apache.hadoop.hbase/, it only took 29 seconds.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: hbasetests.sh
shell cript to execute the tests:
- small in // within the jvm
- medium & large with 2 jvm
The jira containts 3 patchs:
- 1 for the pom
- 1 for the java modifs
- 1 for the shell script
I will create another Jira when I commit the impact for surefire.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144340#comment-13144340 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Using the shell script on MacBook I got:
{code}
Failed tests: testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication): Waited too much time for put replication
queueFailover(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for queueFailover replication
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 142.877 sec <<< FAILURE!
{code}
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.patch
The patch on pom.xml creates 3 profiles:
- nonParallelTests, the default, same behaviour as today: fork always & no //
- parallelTests: 3 threads within a jvm, forked once
- singleJVMTests: fork once, no //. Not used today, but can be useful for tests
To test on a separate bix to see if there is no side effect with the default box. At the beginning everything was going on the console without clear reason.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143316#comment-13143316 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502176/20111103_4737_pom.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/157//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/157//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145145#comment-13145145 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502689/4737_all.patch
against trunk revision .
-1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
+1 tests included. The patch appears to include 376 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 48 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.master.TestMasterFailover
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/198//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/198//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/198//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144235#comment-13144235 ]
nkeywal commented on HBASE-4737:
--------------------------------
@ted from the comment in HBASE-4746: "it already allows to run unit tests in parallel in 10 minutes. A fix for the trunk will follow.", it seems that it should work quite well at the end!
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144711#comment-13144711 ]
Ted Yu commented on HBASE-4737:
-------------------------------
I think the behavior on Mac is related to the handling of ps command.
Let's tackle that first.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: hbasetests.sh
The script launch first the small tests (list hardcoded until the categories are set in the source code) and run the other tests with two parallel maven instance.
Can replay the failed tests.
Still beta, but improve the build time by around 40% => less than 1 hour for the whole suite test.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144224#comment-13144224 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502487/hbasetests.sh
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/176//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.v2.patch
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146755#comment-13146755 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Integrated into TRUNK.
Thanks for the patch N.
hbasetests.sh is added to dev-support/ directory.
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144965#comment-13144965 ]
nkeywal commented on HBASE-4737:
--------------------------------
@Mikael Thank you for the zookeeper patch. TestHRegionInfo.testGetSetOfHTD is not related to the parallel runs, it can fail on an isolated run. I created HBASE-4753 for it.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144190#comment-13144190 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Awesome, N.
It would be nice to see what happens after HBASE-4746 is integrated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144242#comment-13144242 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502469/20111103_4737_pom.v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/174//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/174//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/174//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143250#comment-13143250 ]
Todd Lipcon commented on HBASE-4737:
------------------------------------
The SUREFIRE-329 patch actually lets you annotate by method, not just by class. So if there are some classes that have a couple small mixed with a large, they don't necessarily have to move to a new class
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Jesse Yates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147199#comment-13147199 ]
Jesse Yates commented on HBASE-4737:
------------------------------------
Good stuff guys! Do you think we can update the book as well with how to annotate classes properly and how to run just the short or short and medium tests or just all?
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143397#comment-13143397 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502183/20111103_4737_pom.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestMasterReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/159//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/159//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146708#comment-13146708 ]
Ted Yu commented on HBASE-4737:
-------------------------------
Here is execution time for TestRegionRebalancing on MacBook Pro:
{code}
Running org.apache.hadoop.hbase.TestRegionRebalancing
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.077 sec
{code}
I am not sure how the difference in runtime between Linux and Mac (Windows) should be addressed.
Looks like we should use runtime on Jenkins as the standard.
Also, almost all tests grow along with development.
We should ask developers to observe current runtime after new test is added to an existing test class.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
"Too many open files" everywhere. Let's try again
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143302#comment-13143302 ]
nkeywal commented on HBASE-4737:
--------------------------------
upload (re)done :-)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143337#comment-13143337 ]
nkeywal commented on HBASE-4737:
--------------------------------
surefire conflict? Is there another build on the same machine?
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144885#comment-13144885 ]
Ted Yu commented on HBASE-4737:
-------------------------------
@Mikhail:
Thanks for sharing your experience.
Can you disclose the number of nodes in the cluster where the map-reduce test runner executes ?
I wonder if this model can be applied to Apache Jenkins where build machines are shared among many projects.
I don't see HBASE-4767. Maybe you were referring to HBASE-4746.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.patch
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145136#comment-13145136 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502696/hbasetests.sh
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/199//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Attachment: 20111103_4737_pom.patch
same file, new test
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144033#comment-13144033 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502387/20111103_4737_pom.dummy.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.util.TestFSUtils
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.master.TestMasterFailover
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/171//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/171//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/171//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146710#comment-13146710 ]
nkeywal commented on HBASE-4737:
--------------------------------
Yes, I used Jenkins time to do the split. But there are differences between the run, and for some tests it can be 100%. So there is some arbitrary decision at the end. RegionRebalancing took 58s on the Jenkins built I use a week ago. A month ago it was taking 46s. In this case, I put it in large because it was identified as 'flaky' in the list published by Ram around 3 weeks ago.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147472#comment-13147472 ]
Lars Hofhansl commented on HBASE-4737:
--------------------------------------
It seems some printlns sneaked into TestSplitTransaction.java:
{code}
- static {
- System.out.println("AAAA static");
- }
@Before public void setup() throws IOException {
- System.out.println("AAAA setup");
...
- System.out.println("AAAA setup createRegion");
...
- System.out.println("AAAA setup ends");
}
...
@Test public void testFailAfterPONR() throws IOException, KeeperException {
- System.out.println("AAAA testFailAfterPONR");
{code}
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144700#comment-13144700 ]
Ted Yu commented on HBASE-4737:
-------------------------------
By not included I refer to TestMasterReplication#testSimplePutDelete (my comment @ 04/Nov/11 21:05)
Maybe it was in the long list of 227 tests which were declared not finished. But that is mis-categorized.
Actually for TRUNK, there shouldn't be so many tests that fail. Maybe we should integrate HBASE-4746 first ?
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146780#comment-13146780 ]
Hudson commented on HBASE-4737:
-------------------------------
Integrated in HBase-TRUNK #2423 (See [https://builds.apache.org/job/HBase-TRUNK/2423/])
HBASE-4737 Categorize the tests into small/medium/large; allow small tests to be run
in parallel within a single JVM (N Keywal)
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/dev-support/hbasetests.sh
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/LargeTests.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/MediumTests.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/SmallTests.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestGlobalMemStoreSize.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/avro/TestAvroServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMultipleTimestamps.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestColumnRangeFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRollingRestart.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRpcMetrics.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestGzipFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestMultiRowResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestRowResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithFilters.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestTransform.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/TestVersionResource.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestDefaultEnvironmentEdge.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestEnvironmentEdgeManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestIdLock.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestIncrementingEnvironmentEdge.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146700#comment-13146700 ]
Ted Yu commented on HBASE-4737:
-------------------------------
4737_all.patch didn't produce regression in tests.
I am fine with integrating it to TRUNK.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Open (was: Patch Available)
seems that the v2 is ok
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4737:
---------------------------
Status: Patch Available (was: Open)
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch, 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144114#comment-13144114 ]
stack commented on HBASE-4737:
------------------------------
@N I'm having similar issue over in HBASE-4553. Trying to figure what it is....
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143290#comment-13143290 ]
stack commented on HBASE-4737:
------------------------------
Weird that its 'too many files' because ulimit printed out at start of test says:
asf001.sp2.ygridcore.net
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2048
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
32768
Running in Jenkins mode
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Categorize the tests into
small/medium/large; allow small tests to be run in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147340#comment-13147340 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503120/4737_part2.v1.patch
against trunk revision .
-1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
+1 tests included. The patch appears to include 359 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 51 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.replication.TestReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/219//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/219//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/219//console
This message is automatically generated.
> Categorize the tests into small/medium/large; allow small tests to be run in parallel within a single JVM
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 20111103_4737_pom.dummy.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.patch, 20111103_4737_pom.v2.patch, 20111103_4737_pom.v2.patch, 4737_all.patch, 4737_all.v2.patch, 4737_part2.v1.patch, LargeTests.java, MediumTests.java, SmallTests.java, hbasetests.sh, hbasetests.sh
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in
small/medium/large; allow small tests to be ran in parallel within a single
JVM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143271#comment-13143271 ]
Hadoop QA commented on HBASE-4737:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502170/20111103_4737_pom.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestOpenedRegionHandler
org.apache.hadoop.hbase.filter.TestColumnRangeFilter
org.apache.hadoop.hbase.io.TestHalfStoreFileReader
org.apache.hadoop.hbase.client.TestMultipleTimestamps
org.apache.hadoop.hbase.mapred.TestTableInputFormat
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
org.apache.hadoop.hbase.master.TestCatalogJanitor
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.rest.TestStatusResource
org.apache.hadoop.hbase.filter.TestFilterList
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.master.TestHMasterRPCException
org.apache.hadoop.hbase.rest.TestMultiRowResource
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.util.TestHBaseFsckComparator
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge
org.apache.hadoop.hbase.io.TestHbaseObjectWritable
org.apache.hadoop.hbase.executor.TestExecutorService
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex
org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler
org.apache.hadoop.hbase.regionserver.TestParallelPut
org.apache.hadoop.hbase.regionserver.TestResettingCounters
org.apache.hadoop.hbase.zookeeper.TestZKTable
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.master.TestDefaultLoadBalancer
org.apache.hadoop.hbase.client.TestHTablePool
org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer
org.apache.hadoop.hbase.regionserver.TestAtomicOperation
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.rest.TestScannerResource
org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel
org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.rest.model.TestScannerModel
org.apache.hadoop.hbase.regionserver.TestScanWithBloomError
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.rest.model.TestTableRegionModel
org.apache.hadoop.hbase.io.hfile.TestHFile
org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker
org.apache.hadoop.hbase.regionserver.TestStoreFile
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
org.apache.hadoop.hbase.master.TestMasterTransitions
org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg
org.apache.hadoop.hbase.util.TestMergeTool
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.util.TestBytes
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
org.apache.hadoop.hbase.rest.TestGzipFilter
org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1
org.apache.hadoop.hbase.rest.model.TestTableSchemaModel
org.apache.hadoop.hbase.rest.model.TestVersionModel
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite
org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel
org.apache.hadoop.hbase.filter.TestParseFilter
org.apache.hadoop.hbase.rest.model.TestCellSetModel
org.apache.hadoop.hbase.TestMultiVersions
org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer
org.apache.hadoop.hbase.filter.TestColumnPrefixFilter
org.apache.hadoop.hbase.security.TestUser
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.client.TestTimestampsFilter
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
org.apache.hadoop.hbase.regionserver.TestMasterAddressManager
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.master.TestSplitLogManager
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.filter.TestDependentColumnFilter
org.apache.hadoop.hbase.client.TestShell
org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking
org.apache.hadoop.hbase.regionserver.TestHRegionInfo
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
org.apache.hadoop.hbase.rest.TestTransform
org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.rest.TestVersionResource
org.apache.hadoop.hbase.util.TestRegionSplitCalculator
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.regionserver.TestMemStore
org.apache.hadoop.hbase.rest.TestTableResource
org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner
org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter
org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary
org.apache.hadoop.hbase.coprocessor.TestWALObserver
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.TestFSTableDescriptorForceCreation
org.apache.hadoop.hbase.client.TestHTableUtil
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.rest.TestRowResource
org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
org.apache.hadoop.hbase.io.hfile.TestSeekTo
org.apache.hadoop.hbase.thrift.TestThriftServer
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
org.apache.hadoop.hbase.util.TestFSUtils
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager
org.apache.hadoop.hbase.regionserver.TestKeepDeletes
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
org.apache.hadoop.hbase.master.TestActiveMasterManager
org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker
org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
org.apache.hadoop.hbase.master.TestLogsCleaner
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.client.TestHCM
org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter
org.apache.hadoop.hbase.io.hfile.TestHFileSeek
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.master.TestClockSkewDetection
org.apache.hadoop.hbase.TestGlobalMemStoreSize
org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.regionserver.TestGetClosestAtOrBefore
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
org.apache.hadoop.hbase.regionserver.TestMinVersions
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.regionserver.TestStoreScanner
org.apache.hadoop.hbase.rest.TestScannersWithFilters
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
org.apache.hadoop.hbase.regionserver.TestStore
org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
org.apache.hadoop.hbase.rest.model.TestCellModel
org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy
org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
org.apache.hadoop.hbase.util.TestKeying
org.apache.hadoop.hbase.regionserver.TestRpcMetrics
org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache
org.apache.hadoop.hbase.regionserver.TestQueryMatcher
org.apache.hadoop.hbase.rest.client.TestRemoteTable
org.apache.hadoop.hbase.catalog.TestCatalogTracker
org.apache.hadoop.hbase.rest.model.TestRowModel
org.apache.hadoop.hbase.ipc.TestDelayedRpc
org.apache.hadoop.hbase.filter.TestFilter
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
org.apache.hadoop.hbase.rest.TestSchemaResource
org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.rest.model.TestTableInfoModel
org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener
org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler
org.apache.hadoop.hbase.util.TestPoolMap
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.regionserver.TestBlocksRead
org.apache.hadoop.hbase.regionserver.TestScanner
org.apache.hadoop.hbase.filter.TestBitComparator
org.apache.hadoop.hbase.replication.TestMasterReplication
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
org.apache.hadoop.hbase.util.TestIdLock
org.apache.hadoop.hbase.regionserver.wal.TestHLog
org.apache.hadoop.hbase.regionserver.TestCompaction
org.apache.hadoop.hbase.avro.TestAvroServer
org.apache.hadoop.hbase.regionserver.TestWideScanner
org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster
org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue
org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.master.TestMasterStatusServlet
org.apache.hadoop.hbase.regionserver.TestRSStatusServlet
org.apache.hadoop.hbase.client.TestMetaScanner
org.apache.hadoop.hbase.regionserver.TestKeyValueHeap
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
org.apache.hadoop.hbase.util.TestCompressionTest
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.regionserver.TestCompactSelection
org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel
org.apache.hadoop.hbase.TestHBaseTestingUtility
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.io.hfile.slab.TestSlab
org.apache.hadoop.hbase.rest.model.TestTableListModel
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.util.TestRegionSplitter
org.apache.hadoop.hbase.io.hfile.TestReseekTo
org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner
org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache
org.apache.hadoop.hbase.TestInfoServers
org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture
org.apache.hadoop.hbase.regionserver.TestSeekOptimizations
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/156//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/156//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/156//console
This message is automatically generated.
> Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-4737
> URL: https://issues.apache.org/jira/browse/HBASE-4737
> Project: HBase
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 20111103_4737_pom.patch
>
>
> 1) Split the tests in 3 categories
> - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM
> - medium: <45s, no flaky, useful to detect bugs immediatly
> - large: remaining
>
> 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch
> - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329
> Small is the default. All other tests will have to be marked Medium or Large with a JUnit category.
> Proposed split:
> Small:122 classes, 479 methods, ~3 minutes when no //)
> Medium: 78 classes, 373 methods, ~23 minutes
> Large: 34 classes, 221 methods, ~60 minutes
> I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA).
> MEDIUM LIST (name; number of methods, time)
> org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468
> org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888
> org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157
> org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762
> org.apache.hadoop.hbase.client.TestHCM 3 21.961
> org.apache.hadoop.hbase.client.TestHTablePool 18 26.274
> org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997
> org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629
> org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365
> org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077
> org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834
> org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346
> org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874
> org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923
> org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596
> org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183
> org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373
> org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045
> org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
> org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
> org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
> org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343
> org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391
> org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724
> org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17
> org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953
> org.apache.hadoop.hbase.master.TestMaster 1 18.918
> org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57
> org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979
> org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675
> org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887
> org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
> org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
> org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764
> org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
> org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
> org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
> org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028
> org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031
> org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087
> org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062
> org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort 1 22.545
> org.apache.hadoop.hbase.regionserver.wal.TestWALReplay 5 29.678
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink 5 23.19
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager 1 0.918
> org.apache.hadoop.hbase.replication.TestReplicationPeer 1 25.518
> org.apache.hadoop.hbase.rest.client.TestRemoteAdmin 1 16.141
> org.apache.hadoop.hbase.rest.client.TestRemoteTable 3 17.827
> org.apache.hadoop.hbase.rest.TestGzipFilter 3 17.404
> org.apache.hadoop.hbase.rest.TestMultiRowResource 3 17.314
> org.apache.hadoop.hbase.rest.TestRowResource 11 17.73
> org.apache.hadoop.hbase.rest.TestScannerResource 5 26.868
> org.apache.hadoop.hbase.rest.TestScannersWithFilters 10 20.774
> org.apache.hadoop.hbase.rest.TestSchemaResource 2 22.441
> org.apache.hadoop.hbase.rest.TestStatusResource 2 14.884
> org.apache.hadoop.hbase.rest.TestTableResource 8 25.658
> org.apache.hadoop.hbase.rest.TestTransform 1 15.966
> org.apache.hadoop.hbase.rest.TestVersionResource 7 19.124
> org.apache.hadoop.hbase.TestGlobalMemStoreSize 1 18.154
> org.apache.hadoop.hbase.TestInfoServers 2 14.9
> org.apache.hadoop.hbase.TestMultiVersions 3 27.945
> org.apache.hadoop.hbase.thrift.TestThriftServer 1 20.072
> org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge 1 0.055
> org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager 2 0.215
> org.apache.hadoop.hbase.util.TestFSUtils 2 19.988
> org.apache.hadoop.hbase.util.TestHBaseFsck 6 41.133
> org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge 1 0.054
> org.apache.hadoop.hbase.util.TestMergeTable 1 27.125
> org.apache.hadoop.hbase.util.TestRegionSplitter 4 28.845
> org.apache.hadoop.hbase.zookeeper.TestZKTable 1 0.529
> org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker 2 0.569
> LARGE
> org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
> org.apache.hadoop.hbase.client.TestAdmin 33 369.833
> org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
> org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
> org.apache.hadoop.hbase.client.TestScannerTimeout 4 47.309
> org.apache.hadoop.hbase.client.TestShell 1 69.65
> org.apache.hadoop.hbase.mapred.TestTableInputFormat 6 24.39
> org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
> org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles 6 35.25
> org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
> org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
> org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed 1 42.761
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
> org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
> org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable 1 20.86
> org.apache.hadoop.hbase.master.TestMasterTransitions 3 22.043
> org.apache.hadoop.hbase.master.TestRestartCluster 2 36.325
> org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
> org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction 1 40.935
> org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad 1 47.073
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster 4 43.359
> org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
> org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
> org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
> org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
> org.apache.hadoop.hbase.replication.TestMultiSlaveReplication 1 47.873
> org.apache.hadoop.hbase.replication.TestReplication 7 163.158
> org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
> org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
> org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
> org.apache.hadoop.hbase.TestZooKeeper 7 86.267
> org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> SMALL
> Everyting else
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira