You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Oliver B. Fischer" <ma...@swe-blog.net> on 2012/01/03 23:27:01 UTC

Performance measurement framework for Mathout

Hi,

I am working for some time on Thotti. Thotti aims to be a performance 
measurement framework for Mahout. It allows you to monitor the 
performance of Mahout and compare different setups (JVM settings, Mahout 
settings).

Thotti allows you to define your own test cases with normal Java classes 
and annotations in a very similar manner to TestNG and similar 
frameworks. At the moment only non-distributed tests can be executed, 
but it is planned to suppport distributed tests too.

For test execution Thotti utilizes currently EC2 instances and contains 
a component to manage EC2 instances (creation, termination). It also 
makes heavy use of S3 to store distribute test, test data and test 
result. But with a little bit work it can be extended to support 
different cloud services or local servers.

Since Thotti is now stable enough for non-distributed tests I would like 
to implement a reference test suite for Mahout for non-distributed 
algorithms.

To build this reference test suite I need your help. Please send me your 
test cases. Thotti is able to run the same test multiple times, with 
different JVM settings and differerent parameters. So you can send me 
your test cases and test data along with different test setups.

The example test case below will be executed by Thotti once. The JVM 
will run with -server.

public class SimpleRecommenderTest {
     @NDTest(id = "BForJVM909",
             run = @Run(jvmArgs = @JVMArgs(id = "jvm909", value = 
"-server")))
     public void executeTest() throws IOException, TasteException {
         DataModel model = new FileDataModel(prependDataDir(new 
File("intro.csv")));

         UserSimilarity similarity = new 
PearsonCorrelationSimilarity(model);

         UserNeighborhood neighborhood =
                 new NearestNUserNeighborhood(2, similarity, model);

         Recommender recommender = new GenericUserBasedRecommender(
                 model, neighborhood, similarity);

         recommender.recommend(1, 1);
     }
}

I would be gratefull for your support on this work.

Bye,

Oliver

Re: Performance measurement framework for Mathout

Posted by Oliver Fischer <ma...@swe-blog.net>.
Of course. As written before it allows you to define test scenarios for 
Mahout as for example: Run this recommender with different JVM settings 
or different parameters. The test scenario will be executed once of each 
given JVM settings and for each given parameter set.

Using this feature you are able to compare the runtime of Mahout for 
different JVM settings and the best configuration for a given algorithm 
from the view point of runtime. For example will be able to compare the 
runtime of your recommander for 10, 50 and 100 recommendation using the 
same test. Thotti collect the results of your tests and visualize them. 
Although the visualization is not implemented yet.

Beyond Thotti can be used even to compare the runtime of Mahout for 
different hardware setups. If you would like to know if Mahout will run 
faster an an different machine setup, run the same tests on the old 
configuration and the new one.

Bye,

Oliver

Am 07.01.2012 08:59, schrieb Isabel Drost:
> On 03.01.2012 Oliver B. Fischer wrote:
>> Thotti's home is at https://github.com/obfischer/repositories
>>
>> I will update the example project
>> (https://github.com/obfischer/Thotti-E2) during the next days. There is
>> also need for some documentation.
>
> Thanks for sharing the links, Oliver. It would be great if you could give a very
> brief overview of what people can do with the framework. Would be great if that
> would develop into something that can be run regularly to avoid performance
> regressions in our code.
>
> Isabel

Re: Performance measurement framework for Mathout

Posted by Isabel Drost <is...@apache.org>.
On 03.01.2012 Oliver B. Fischer wrote:
> Thotti's home is at https://github.com/obfischer/repositories
> 
> I will update the example project
> (https://github.com/obfischer/Thotti-E2) during the next days. There is
> also need for some documentation.

Thanks for sharing the links, Oliver. It would be great if you could give a very 
brief overview of what people can do with the framework. Would be great if that 
would develop into something that can be run regularly to avoid performance 
regressions in our code.

Isabel

Re: Performance measurement framework for Mathout

Posted by "Oliver B. Fischer" <ma...@swe-blog.net>.
Hello Keren,

Thotti's home is at https://github.com/obfischer/repositories

I will update the example project 
(https://github.com/obfischer/Thotti-E2) during the next days. There is 
also need for some documentation.

Bye

Oliver


Am 03.01.2012 23:41, schrieb Keren Ouaknine:
> Hello Oliver,
>
> It sounds like a great project, can you send me a link with more
> information?
>
> I dont have new use cases, as I am using Mahout's *samples* for test
> performance in front of other systems.
> Thanks,
> Keren
>
> On Tue, Jan 3, 2012 at 5:27 PM, Oliver B. Fischer<ma...@swe-blog.net>wrote:
>
>> Hi,
>>
>> I am working for some time on Thotti. Thotti aims to be a performance
>> measurement framework for Mahout. It allows you to monitor the performance
>> of Mahout and compare different setups (JVM settings, Mahout settings).
>>
>> Thotti allows you to define your own test cases with normal Java classes
>> and annotations in a very similar manner to TestNG and similar frameworks.
>> At the moment only non-distributed tests can be executed, but it is planned
>> to suppport distributed tests too.
>>
>> For test execution Thotti utilizes currently EC2 instances and contains a
>> component to manage EC2 instances (creation, termination). It also makes
>> heavy use of S3 to store distribute test, test data and test result. But
>> with a little bit work it can be extended to support different cloud
>> services or local servers.
>>
>> Since Thotti is now stable enough for non-distributed tests I would like
>> to implement a reference test suite for Mahout for non-distributed
>> algorithms.
>>
>> To build this reference test suite I need your help. Please send me your
>> test cases. Thotti is able to run the same test multiple times, with
>> different JVM settings and differerent parameters. So you can send me your
>> test cases and test data along with different test setups.
>>
>> The example test case below will be executed by Thotti once. The JVM will
>> run with -server.
>>
>> public class SimpleRecommenderTest {
>>     @NDTest(id = "BForJVM909",
>>             run = @Run(jvmArgs = @JVMArgs(id = "jvm909", value =
>> "-server")))
>>     public void executeTest() throws IOException, TasteException {
>>         DataModel model = new FileDataModel(prependDataDir(**new
>> File("intro.csv")));
>>
>>         UserSimilarity similarity = new PearsonCorrelationSimilarity(**
>> model);
>>
>>         UserNeighborhood neighborhood =
>>                 new NearestNUserNeighborhood(2, similarity, model);
>>
>>         Recommender recommender = new GenericUserBasedRecommender(
>>                 model, neighborhood, similarity);
>>
>>         recommender.recommend(1, 1);
>>     }
>> }
>>
>> I would be gratefull for your support on this work.
>>
>> Bye,
>>
>> Oliver
>>
>>
>
>

Re: Performance measurement framework for Mathout

Posted by Keren Ouaknine <ke...@gmail.com>.
Hello Oliver,

It sounds like a great project, can you send me a link with more
information?

I dont have new use cases, as I am using Mahout's *samples* for test
performance in front of other systems.
Thanks,
Keren

On Tue, Jan 3, 2012 at 5:27 PM, Oliver B. Fischer <ma...@swe-blog.net>wrote:

> Hi,
>
> I am working for some time on Thotti. Thotti aims to be a performance
> measurement framework for Mahout. It allows you to monitor the performance
> of Mahout and compare different setups (JVM settings, Mahout settings).
>
> Thotti allows you to define your own test cases with normal Java classes
> and annotations in a very similar manner to TestNG and similar frameworks.
> At the moment only non-distributed tests can be executed, but it is planned
> to suppport distributed tests too.
>
> For test execution Thotti utilizes currently EC2 instances and contains a
> component to manage EC2 instances (creation, termination). It also makes
> heavy use of S3 to store distribute test, test data and test result. But
> with a little bit work it can be extended to support different cloud
> services or local servers.
>
> Since Thotti is now stable enough for non-distributed tests I would like
> to implement a reference test suite for Mahout for non-distributed
> algorithms.
>
> To build this reference test suite I need your help. Please send me your
> test cases. Thotti is able to run the same test multiple times, with
> different JVM settings and differerent parameters. So you can send me your
> test cases and test data along with different test setups.
>
> The example test case below will be executed by Thotti once. The JVM will
> run with -server.
>
> public class SimpleRecommenderTest {
>    @NDTest(id = "BForJVM909",
>            run = @Run(jvmArgs = @JVMArgs(id = "jvm909", value =
> "-server")))
>    public void executeTest() throws IOException, TasteException {
>        DataModel model = new FileDataModel(prependDataDir(**new
> File("intro.csv")));
>
>        UserSimilarity similarity = new PearsonCorrelationSimilarity(**
> model);
>
>        UserNeighborhood neighborhood =
>                new NearestNUserNeighborhood(2, similarity, model);
>
>        Recommender recommender = new GenericUserBasedRecommender(
>                model, neighborhood, similarity);
>
>        recommender.recommend(1, 1);
>    }
> }
>
> I would be gratefull for your support on this work.
>
> Bye,
>
> Oliver
>
>


-- 
Keren Ouaknine
Web: www.kereno.com