You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2011/11/01 09:24:32 UTC
[jira] [Created] (HBASE-4712) Document rules for writing tests
Document rules for writing tests
--------------------------------
Key: HBASE-4712
URL: https://issues.apache.org/jira/browse/HBASE-4712
Project: HBase
Issue Type: Task
Components: test
Affects Versions: 0.92.0
Reporter: nkeywal
Priority: Minor
We saw that some tests could be improved. Documenting the general rules could help.
Proposal:
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
Small and medium tests must not need more than 30 minutes to run altogether.
Small and medium tests should be executed by the developers before submitting a patch.
Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
- All tests should be written to be as fast as possible.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
- Sleeps:
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
- Tests using cluster:
- Tests using a HRegion do not have to start a cluster: A region can use the local file system.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153609#comment-13153609 ]
stack commented on HBASE-4712:
------------------------------
This is excellent. Thanks for writing it up Nicolas. I believe it belongs in this chapter: http://hbase.apache.org/book.html#developer I can get it. Lets leave it hang out here a little while longer in case there are comments after your email today.
Do you think the long tests play into HBASE-4821? Or at least, we should be able to run the long tests in whatever comes of hbase-4812.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165039#comment-13165039 ]
Hudson commented on HBASE-4712:
-------------------------------
Integrated in HBase-TRUNK-security #25 (See [https://builds.apache.org/job/HBase-TRUNK-security/25/])
HBASE-4712 Document rules for writing tests; Addendum via Jesse Yates
stack :
Files :
* /hbase/trunk/src/docbkx/developer.xml
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt, test-doc-cleanup.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164873#comment-13164873 ]
Hudson commented on HBASE-4712:
-------------------------------
Integrated in HBase-TRUNK #2525 (See [https://builds.apache.org/job/HBase-TRUNK/2525/])
HBASE-4712 Document rules for writing tests; Addendum via Jesse Yates
stack :
Files :
* /hbase/trunk/src/docbkx/developer.xml
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt, test-doc-cleanup.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143320#comment-13143320 ]
nkeywal commented on HBASE-4712:
--------------------------------
For 15s, it's fuzzy to help parallelization. But there are only a few tests that don't use a cluster and last more than 15s, I could give them a try.
org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818
org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226
org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694
org.apache.hadoop.hbase.regionserver.TestMemStore 21 28.672
org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
org.apache.hadoop.hbase.util.TestIdLock 1 15.198
Same for 50, and as well the longer it last, the more chance you have to get a flaky test :-)
The tests over 50s are:
org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463
org.apache.hadoop.hbase.client.TestAdmin 33 369.833
org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41
org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497
org.apache.hadoop.hbase.client.TestShell 1 69.65
org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce 1 92.144
org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606
org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658
org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738
org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379
org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537
org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346
org.apache.hadoop.hbase.replication.TestReplication 7 163.158
org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742
org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654
org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25
org.apache.hadoop.hbase.TestZooKeeper 7 86.267
org.apache.hadoop.hbase.util.TestMergeTool 1 257.223
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Jesse Yates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161470#comment-13161470 ]
Jesse Yates commented on HBASE-4712:
------------------------------------
bq. As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
It would be cool if we could actually do that with the annotation. For instance:
@LargeTest(modifications=false)
would be a large test that hasn't made modifications to the minicluster. Makes it really easy to parallelize in the same jvm later. Another component I could see needing for the same jvm minicluster is a getUniqueTable() method. But lets put that off until we actually get there :)
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4712) Document rules for writing tests
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4712:
-------------------------
Attachment: test-doc-cleanup.txt
Corrections from Jesse Yates
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt, test-doc-cleanup.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143298#comment-13143298 ]
stack commented on HBASE-4712:
------------------------------
Why the 15 and 50 second limits. Are they fuzzy limits? If hard, why these numbers?
Otherwise, excellent.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164279#comment-13164279 ]
Hudson commented on HBASE-4712:
-------------------------------
Integrated in HBase-TRUNK-security #24 (See [https://builds.apache.org/job/HBase-TRUNK-security/24/])
HBASE-4712 Document rules for writing tests
stack :
Files :
* /hbase/trunk/src/docbkx/developer.xml
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143323#comment-13143323 ]
stack commented on HBASE-4712:
------------------------------
Grand.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal reassigned HBASE-4712:
------------------------------
Assignee: nkeywal
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153539#comment-13153539 ]
nkeywal commented on HBASE-4712:
--------------------------------
New proposal:
1) Running tests
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
- Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly (the maximum execution time for a test is 15 seconds and they do not use a cluster) in a shared jvm.
- Medium tests represents tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They're designed to less than 50 seconds. They can use a cluster, and each of them are executed in a separate JVM.
- Large tests are everything else. They are typically integration tests, regression tests for specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed before a commit on the pre-integration machines. They can be run on the developper as well.
Commands are:
- mvn test - execute all tests in a separate JVM. All results are presented in a single report.
- mvn -P runDevTests - execute small tests in a single JVM and medium tests in separates jvm. There are two reports.
- mvn -P runAllTests - execute small tests in a single JVM and medium and large tests in separates jvm. There are two reports, one for small, one for medium and large.
- mvn -P runSmallTests - execute small tests in a single JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large tests in parallel with two maven instances, and provide a single report. It must be executed from the directory which contains the pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.
- Categories and executin time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good parallelisation for the ones using it, and ease the analysis when the test fails.
- Sleeps:
- Whenever possible, tests should not use sleep, but rather waiting for the real event. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
- Tests using cluster:
- Tests using a HRegion do not have to start a cluster: A region can use the local file system.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
@committers: Please tell me where I should put this, I will put it as a patch (with the new comments taken into account of course!)
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141281#comment-13141281 ]
stack commented on HBASE-4712:
------------------------------
@N You've written it as a proposal which is good. Float it on dev list and get some edits then we'll commit what folks agree on. Good stuff.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161499#comment-13161499 ]
nkeywal commented on HBASE-4712:
--------------------------------
updated version after Jesse's comments:
--
1) Running tests
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
This appears like this:
{noformat}
@Category(SmallTests.class)
public class TestHRegionInfo {
@Test
public void testCreateHRegionInfoName() throws Exception {
// ...
}
}
{noformat}
Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly (the maximum execution time for a test is 15 seconds, and they do not use a cluster) in a shared jvm.
Medium tests represents tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They're designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM.
Large tests are everything else. They are typically integration tests, regression tests for specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well.
Commands are:
1.1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there are two reports, one report for small tests, and one report for medium tests if they are executed
1.2) mvn test -P runAllTests
- execute small tests in a single JVM then medium and large tests in a separate JVM for each test.
- medium and large tests are NOT executed if there is an error in a small test
- there are two reports, one report for small tests, and one report for medium and large tests if they are executed
1.3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are taken into account)
- use actually the official release of surefire & the old connector to junit
- tests are executed in separated JVM
- you will see a new message at the end of the report: "[INFO] Tests are skipped". It's harmless.
1.4) Various other profiles
- mvn test -P runSmallTests - execute small tests only, in a single JVM.
- mvn test -P runMediumTests - execute medium tests in a separate JVM.
- mvn test -P runLargeTests - execute large tests in a separate JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large tests in parallel with two maven instances, and provide a single report. It must be executed from the directory which contains the pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.
Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good parallelization for people using it, and ease the analysis when the test fails.
Sleeps
- Whenever possible, tests should not use Thread.sleep, but rather waiting for the real event they need. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
Tests using a cluster
- Tests using a HRegion do not have to start a cluster: A region can use the local file system instead of a distributed one.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per test class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161498#comment-13161498 ]
nkeywal commented on HBASE-4712:
--------------------------------
bq. Is that one report each for the medium and large or one report for the two? Just need a single word of clarification in there
One report for the two. I will reformulate this part.
bq. Is this new behavior from what's in trunk currently? Because I'm pretty sure I can just run mvn test -Dtest=myTest and get it to run them.
Yes, it would change from what we have in trunk today. This profile deactivates categories, if not re they are still taken into account even if you specify "-Dtest=myTest", so if your test is in largeTests it's not executed.
bq. I don't think we really want to have these options available, right? A lot of the large tests depend on the fact that jvms are forked.
My bad, copy/paste error, I am going to fix it
bq. A simple example or two here with the annotations would make this super obvious. For example:
Ok, will add an example.
bq. Do we need to add a MiniRegion just for testing these situations?
It runs like that out of the box (the region can use the local system), see for example TestScanner.
bq. It would be cool if we could actually do that with the annotation. For instance: @LargeTest(modifications=false)
If we can't check automatically that the annotation "does not lie", we have the risk of having to look at the source code to check if it's true. As well, we can have test with a default cluster but that cannot run with other tests (if they try to timout the master for example), so I believe we will have to be explicit (something like sharedCluster=true). Note as well that a test can belong to multiple categories.
Thanks for the review!
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153612#comment-13153612 ]
stack commented on HBASE-4712:
------------------------------
On commit, add to the doc this note from mail on list:
bq. By default there all executed in different JVM, and the second report is skipped. So you will see a new message at the end of the report: "[INFO] Tests are skipped". It's harmless."
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160415#comment-13160415 ]
stack commented on HBASE-4712:
------------------------------
I am +1 on this patch/doc. Will commit in a while (will ping Jesse first to make sure he is cool w/ it). Jesse's IntegrationTest* stuff in failsafe is something distinct. We can add a section later making it clear the diff between this work here and Jesse's.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Jesse Yates (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161464#comment-13161464 ]
Jesse Yates commented on HBASE-4712:
------------------------------------
Couple of comments:
bq. there is one report for small tests and one report for medium and large tests if they are executed
Is that one report each for the medium and large or one report for the two? Just need a single word of clarification in there
bq. 1.3) mvn test -P localTests -Dtest=myTests
Is this new behavior from what's in trunk currently? Because I'm pretty sure I can just run
mvn test -Dtest=myTest and get it to run them.
{quote}
mvn test -P runMediumTests - execute medium tests in a single JVM.
mvn test -P runLargeTests - execute medium tests in a single JVM.
{quote}
I don't think we really want to have these options available, right? A lot of the large tests depend on the fact that jvms are forked.
bq. Categories and execution time
A simple example or two here with the annotations would make this super obvious. For example:
{quote}
a small test would look like this:
<code>
@SmallTest
public class TestMyClass{
...
}
</code>
Or you could test each method:
public class TestMyClass{
@SmallTest
public void testMyMethod() ...
}
</code>
{quote}
This is assuming the per-method annotation still work as Todd suggested a while ago?
bq. Tests using a HRegion do not have to start a cluster: A region can use the local file system.
Do we need to add a MiniRegion just for testing these situations?
Otherwise, I'm fine with the rest of the comments.
However, with recent discussions(http://search-hadoop.com/m/q41O6YiyfN), there may be some changes coming to this soon.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4712) Document rules for writing tests
Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4712.
--------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
Hadoop Flags: Reviewed
Committed to trunk. Thanks for the doc. N.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160211#comment-13160211 ]
nkeywal commented on HBASE-4712:
--------------------------------
Updated version. I suppose there is a tool to put that in the existing xml. I naively thought that open office would do it, but it's not the case...
--
1) Running tests
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly (the maximum execution time for a test is 15 seconds, and they do not use a cluster) in a shared jvm.
Medium tests represents tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They're designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM.
Large tests are everything else. They are typically integration tests, regression tests for specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well.
Commands are:
1.1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there is one report for small tests, and one report for medium tests if they are executed
1.2) mvn test -P runAllTests
- execute small tests in a single JVM then medium and large tests in a separate JVM for each test.
- medium and large tests are NOT executed if there is an error in a small test
- there is one report for small tests, and one report for medium and large tests if they are executed
1.3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are taken into account)
- use actually the official release of surefire & the old connector to junit
- tests are executed in separated JVM
- you will see a new message at the end of the report: "[INFO] Tests are skipped". It's harmless.
1.4) mvn test -P runAllTests
- execute small tests in a single JVM then medium & large tests in a separate JVM for each test
- medium and large tests are NOT executed if there is an error in a small test
- large tests are NOT executed if there is an error in a small or medium test
- there are one report for small tests, and one report for medium & large tests if they are executed
1.5) Various other profiles
mvn test -P runSmallTests - execute small tests only, in a single JVM.
mvn test -P runMediumTests - execute medium tests in a single JVM.
mvn test -P runLargeTests - execute medium tests in a single JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large tests in parallel with two maven instances, and provide a single report. It must be executed from the directory which contains the pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.
Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good parallelization for people using it, and ease the analysis when the test fails.
Sleeps
- Whenever possible, tests should not use Thread.sleep, but rather waiting for the real event they need. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
Tests using a cluster
- Tests using a HRegion do not have to start a cluster: A region can use the local file system.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per test class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164748#comment-13164748 ]
stack commented on HBASE-4712:
------------------------------
I committed Jesse's addendum to TRUNK.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt, test-doc-cleanup.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158608#comment-13158608 ]
nkeywal commented on HBASE-4712:
--------------------------------
Udpated doc, taking into account the possibility to use mvn verify for large test. Will send it to the list after a sucessful pre-integration (hadoop qa fails often these days).
-
1) Running tests
HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTests, MediumTests, LargeTests.
- Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly (the maximum execution time for a test is 15 seconds, and they do not use a cluster) in a shared jvm.
- Medium tests represents tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They're designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM.
- Large tests are everything else. They are typically integration tests, regression tests for specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well.
Commands are:
1) mvn test
- execute small tests in a single JVM and medium tests in a separate JVM for each test
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed
- there is one report for small tests, and one report for medium tests -if they are executed-
2) mvn verify
- execute small tests in a single JVM then medium tests in a separate JVM for each test, then large tests in a separate JVM as well.
- medium tests are NOT executed if there is an error in a small test
- large tests are NOT executed if there is an error in a small or medium test
- there is one report by test category, small, medium and large
3) mvn test -P localTests -Dtest=myTests
- remove any category effect (without this specific profile, the profiles are taken into account)
- use actually the official release of surefire & the old connector to junit
- tests are executed in separated JVM
- You will see a new message at the end of the report: "[INFO] Tests are skipped". It's harmless.
4) mvn test -P runAllTests
- execute small tests in a single JVM then medium & large tests in a separate JVM for each test
- medium and large tests are NOT executed if there is an error in a small test
- large tests are NOT executed if there is an error in a small or medium test
- there are one report for small tests, and one report for medium & large tests -if they are executed-
5) Various other profiles
- mvn test -P runSmallTests - execute small tests only, in a single JVM.
- mvn test -P runMediumTests - execute medium tests in a single JVM.
- mvn test -P runLargeTests - execute medium tests in a single JVM.
It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large tests in parallel with two maven instances, and provide a single report. It must be executed from the directory which contains the pom.xml. Commands are:
./dev-support/hbasetests.sh - execute small and medium tests
./dev-support/hbasetests.sh runAllTests - execute all tests
./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate jvm and without parallelisation.
2) Writing tests
Tests rules & hints are:
- As most as possible, tests should be written as small tests.
- All tests must be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
- Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
- Tests can be written with HBaseTestingUtility . This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster.
- Categories and execution time
- All tests must be categorized, if not they could be skipped.
- All tests should be written to be as fast as possible.
- Small tests should last less than 15 seconds, and must not have any side effect.
- Medium tests should last less than 45 seconds.
- large tests should last less than 3 minutes, this ensure a good parallelization for the ones using it, and ease the analysis when the test fails.
- Sleeps:
- Whenever possible, tests should not use sleep, but rather waiting for the real event. This is faster and clearer for the reader.
- Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
- Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
- Tests using a cluster:
- Tests using a HRegion do not have to start a cluster: A region can use the local file system.
- Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per test class.
- Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
- As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164013#comment-13164013 ]
Hudson commented on HBASE-4712:
-------------------------------
Integrated in HBase-TRUNK #2522 (See [https://builds.apache.org/job/HBase-TRUNK/2522/])
HBASE-4712 Document rules for writing tests
stack :
Files :
* /hbase/trunk/src/docbkx/developer.xml
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4712.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4712) Document rules for writing tests
Posted by "stack (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4712:
-------------------------
Attachment: 4712.txt
Here is the nkeywal doc hacked into docbook. I also moved stuff around breaking out a 'Tests' subsection under developer chapter and stuck Jesse's integration stuff in here too (with some attempt at lead-in text distingushing the two)
> Document rules for writing tests
> --------------------------------
>
> Key: HBASE-4712
> URL: https://issues.apache.org/jira/browse/HBASE-4712
> Project: HBase
> Issue Type: Task
> Components: test
> Affects Versions: 0.92.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Attachments: 4712.txt
>
>
> We saw that some tests could be improved. Documenting the general rules could help.
> Proposal:
> HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest
> Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster.
> Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally.
> Small and medium tests must not need more than 30 minutes to run altogether.
> Small and medium tests should be executed by the developers before submitting a patch.
> Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests.
> Tests rules & hints are:
> - As most as possible, tests should be written as small tests.
> - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names.
> - All tests should be written to be as fast as possible.
> - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests.
> - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster.
> - Sleeps:
> - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances.
> - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop.
> - Tests using cluster:
> - Tests using a HRegion do not have to start a cluster: A region can use the local file system.
> - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class.
> - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories.
> - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira