[GitHub] incubator-phoenix pull request: Version two for making phoenix tes...

[samarthjain <gi...@git.apache.org>]

GitHub user samarthjain opened a pull request:

    https://github.com/apache/incubator-phoenix/pull/33

    Version two for making phoenix tests run faster

    Following is the overview of changes in this commit:
    
    1) We now have integration tests divided in three categories: HBaseTimeManaged, ClientTimeManaged and TestThatNeedsOwnMiniCluster. 
    
    HBaseTimeManaged and ClientTimeManaged tests use one mini cluster isolated for each category of tests.  We don't need to any longer start and shutdown mini-cluster at the beginning and end of tests. 
    
    Tests categorized as TestThatNeedsOwnMiniCluster spin up their own mini-clusters. 
    
    2) The number of threads used by a mini cluster has been reduced from around 200 to 130. Also, the max heap memory allocated while running integration tests is now reduced to 2GB from 3GB. The two changes let us run more number of forked JVMs running integration tests. For example - on my 24GB 8 core machine, I can easily run up to 10 forked JVMs. Reducing the number of threads used by mini-cluster also helps us in preventing hitting the number of threads limit imposed by OS. 
    
    3) All the tests have been now annotated as belonging to one of the three categories. Once, Junit4.12 is out, we hopefully won't need to add annotations for every new test class. Just assigning the Category annotation to the base test class will be sufficient. 
    
    4) QueryIT has been broken up into various tests. This helps in parallelism as otherwise this test class would alone take around 4min 30 seconds to finish. Breaking up into different sub-classes helped reduce the execution time by around a minute. 
    
    5) Timing numbers:
    On my dev box, running 8 concurrent JVMS using mvn install -DnumForkedIT=8, tests under phoenix-core module finished in approximately 6 min 20 seconds. The same used to take around 50 mins before the optimization work. v1 of faster tests that spun up and down a mini cluster for every test used to take around 10 mins for the phoenix-core module.
    
    The files with the major changes are:
    pom.xml
    PhoenixTestDriver.java
    ConnectionQueryServicesTestImpl.java
    BaseTest.java - a lot of code form BaseConnectedQueryIT.java has been moved here. 
    QueryIT.java
    ClientManagedTimeTest.java
    HBaseManagedTimeTest.java
    NeedsOwnMinitClusterTest.java
    BaseClientManagedTimeIT.java
    BaseHBaseManagedTimeIT.java
    BaseConnectedQueryIT.java - deleted
    Shadower.java - method level annotation to help tell the developers that a public static method is being shadowed
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/samarthjain/incubator-phoenix master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-phoenix/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #33
    
----
commit 6bc4380c38e37a5e2fa0203218498db2c9f49505
Author: samarthjain <sa...@gmail.com>
Date:   2014-05-16T01:58:00Z

    Faster tests v2

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-phoenix pull request: Version two for making phoenix tes...

[jtaylor-sfdc <gi...@git.apache.org>]

Github user jtaylor-sfdc commented on the pull request:

    https://github.com/apache/incubator-phoenix/pull/33#issuecomment-43294166
  
    Excellent! Great work, @samarthjain. Mind opening a JIRA and attaching patches for 3.0 and 4.0 branch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---