You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2011/08/16 21:10:27 UTC
[jira] [Created] (HBASE-4207) Run test suite in parallel, multiple
concurrent test instances.
Run test suite in parallel, multiple concurrent test instances.
---------------------------------------------------------------
Key: HBASE-4207
URL: https://issues.apache.org/jira/browse/HBASE-4207
Project: HBase
Issue Type: Task
Components: test
Reporter: stack
Attachments: parallel.build.txt
>From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4207) Run test suite in parallel,
multiple concurrent test instances.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116770#comment-13116770 ]
stack commented on HBASE-4207:
------------------------------
I learned that this parallel mechanism only works if you do NOT for to run tests: http://stackoverflow.com/questions/3600090/maven-surefire-unable-to-fork-parallel-test-execution/7426894#7426894 My sense is we have to fork to protect against tests that run wild (or if we move the crazies into integrated test category, maybe we could run non-forked).
Trying to build w/o forking, most tests pass. A few fail such as TestTableMapReduce and TestCompaction. Would have to look at these.
When I try w/ parallel on, we are running 1/10th of the tests (but they run fast).
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4207) Run test suite in parallel, multiple
concurrent test instances.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4207:
-------------------------
Attachment: parallel.build.txt
Some pom machinations to add parallel test
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4207) Run test suite in parallel,
multiple concurrent test instances.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085948#comment-13085948 ]
stack commented on HBASE-4207:
------------------------------
Its doing some kinda weird pause every so often. What I have attached seemed to run longer than serial running of tests. Let me mess some more.
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4207) Run test suite in parallel, multiple
concurrent test instances.
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal resolved HBASE-4207.
----------------------------
Resolution: Duplicate
@lars
Yep. There are still stuff to do in this area, but it better to have new jiras.
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4207) Run test suite in parallel,
multiple concurrent test instances.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116809#comment-13116809 ]
stack commented on HBASE-4207:
------------------------------
All the parallel running zks make for interesting logs:
{code}
2011-09-28 20:31:34,151 INFO [pool-1-thread-104] hbase.HBaseTestingUtility(445): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s)
2011-09-28 20:31:34,178 INFO [pool-1-thread-99] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21826
2011-09-28 20:31:34,179 INFO [pool-1-thread-69] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21830
2011-09-28 20:31:34,179 INFO [pool-1-thread-61] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21818
2011-09-28 20:31:34,179 INFO [pool-1-thread-78] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21829
2011-09-28 20:31:34,179 INFO [pool-1-thread-44] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21821
2011-09-28 20:31:34,179 INFO [pool-1-thread-76] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21820
2011-09-28 20:31:34,184 INFO [pool-1-thread-71] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21823
2011-09-28 20:31:34,183 INFO [pool-1-thread-23] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21825
2011-09-28 20:31:34,183 INFO [pool-1-thread-74] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21822
2011-09-28 20:31:34,183 INFO [pool-1-thread-114] zookeeper.MiniZooKeeperCluster(164): Started MiniZK Cluster and connect 1 ZK server on client port: 21831
{code}
I played around with the various attributes and considering all the factors -- no-forking, plugin buggyness (when I change threads, a different subset of the tests run -- never more than 1/4), and the upped complexity of debugging failed tests when they are being run beside others -- I am going to put aside this avenue of pursuit for now.
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4207) Run test suite in parallel,
multiple concurrent test instances.
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440566#comment-13440566 ]
Lars Hofhansl commented on HBASE-4207:
--------------------------------------
N did this, no? I think we can close this one.
> Run test suite in parallel, multiple concurrent test instances.
> ---------------------------------------------------------------
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
> Issue Type: Task
> Components: test
> Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests in parallel. I'm trying it. I'll attach the patch to do classes in parallel (as opposed to methods) with four threads per core max.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira