Performance Tests - Suggestions and Ideas

[Richard Hirsch <hi...@gmail.com>]

I enhanced the performance test page in the wiki
(http://cwiki.apache.org/confluence/display/ESME/Performance+tests)
with more details and test ideas

@Markus: I think it would great if you could do the performance tests
once a week on the Stax performance instance.  Initially, it would be
great just to have test suite that we can run. It is critical that we
can put such tests into our continuous integration cycle but until
this is possible, it would be great if you could just do these short
tests on a weekly basis. That would allow us to make sure that code
changes aren't impacting our performance. I'd like to make these tests
as easy for you to perform as possible.

Although regression testing would be one goal, the changing of small
parameters (adding one action for each user) and seeing the impact on
performance would also be very interesting.

We should probably define some sort of a process that describes the
tests to make sure that the basis is always the same. For example,

1. Drop existing DB - this can be via a stax command
2. Create existing DB  - this can be via a stax command
3. Deploy application - - this can be via a stax command
4. Test - this could be done via maven
5. Take screenshots of Stax console (still have to define which
screens we need - maybe like the ones you sent me last time)
6. Mail all the details to me and I post to wiki

Thoughts?

D.

Re: Performance Tests - Suggestions and Ideas

[David Pollak <fe...@gmail.com>]

On Fri, Nov 20, 2009 at 5:23 AM, Markus Kohler <ma...@gmail.com>wrote:

> Hi all,
> Just as a teaser ;-)
> The textile support is expensive in regards to the memory allocated (freed
> afterwards).
> Details will come as soon as I find some time.
>

Yeah.  Lift's Textile parser using Scala's parser combinator library... it's
low and memory intensive.  Anyone want to sign up to re-write it? ;-)


>
> Regards,
> Markus
>
> On Fri, Nov 20, 2009 at 1:58 PM, Vassil Dichev <vd...@apache.org> wrote:
>
> > I just wanted to say thanks to Markus for doing this, and I think
> > performance testing is a critical aspect for this type of application
> > (as Twitter have demonstrated very convincingly).
> >
> > I also believe performance testing of ESME is also a great way to test
> > Lift, as ESME is one of the biggest open-source projects using Lift
> > (ok, there aren't many anyway). Other bugs which performance tests
> > could catch would be deadlocks and thread contention, and these are
> > vitally important in the light of the dynamic change Lift is
> > undergoing swapping its actor implementations.
> >
> > It's also a good idea to run performance tests on Hudson, at least for
> > the time being- we don't have that many commits per week yet.
> >
> > About memory analysis being a low-hanging fruit: agree 100%, provided
> > one condition is fulfilled. The condition is that we're able to test
> > the application locally, and performance tests help us do exactly
> > that. Previously many bugs could only be reproduced by everyone on the
> > Stax instance. Besides, a long time ago I have worked on memory leak
> > problems in the same area as Markus, so I think we have enough
> > know-how.
> >
> > Vassil
> >
>



-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

Re: Performance Tests - Suggestions and Ideas

[Vassil Dichev <vd...@apache.org>]

> Sure, modern VM's are quite efficient for short lived objects, but they
> still have limits as soon as you go multi threaded. Sun has IMHO send the
> wrong message out in this regard.

Right, if all of your cores wait while the GC is running, that's
tough. The concurrent collector might help a bit.

> Just one number, sending one short message (10 chars ?) caused  70 Mbyte to
> be allocated in net.liftweb.textile.TextileParser$.toHtml(java.lang.String,
> scala.Option, boolean)
>  and  (probably) freed them afterwards. More details later ...

Ugh, that's way too much indeed. Is it just when running the
TextileParser the first time or every time a message is sent? You got
me interested.

Re: Performance Tests - Suggestions and Ideas

[Markus Kohler <ma...@gmail.com>]

Hi Vassil,
Sure, modern VM's are quite efficient for short lived objects, but they
still have limits as soon as you go multi threaded. Sun has IMHO send the
wrong message out in this regard.

Just one number, sending one short message (10 chars ?) caused  70 Mbyte to
be allocated in net.liftweb.textile.TextileParser$.toHtml(java.lang.String,
scala.Option, boolean)
 and  (probably) freed them afterwards. More details later ...


Regards,
Markus

On Fri, Nov 20, 2009 at 2:49 PM, Vassil Dichev <vd...@apache.org> wrote:

> > Just as a teaser ;-)
> > The textile support is expensive in regards to the memory allocated
> (freed
> > afterwards).
> > Details will come as soon as I find some time.
>
> OK, no surprises there. For a single message there might even be more
> than one text area which is parsed by textile, as our own MsgParser
> must process the message first into text, hashtags, users and links.
> As you mentioned, the objects from the parse tree are short-lived
> though so I don't expect them to even leave the eden space on the Sun
> VM. Not sure how big a performance impact this will have.
>

Re: Performance Tests - Suggestions and Ideas

[Vassil Dichev <vd...@apache.org>]

> Just as a teaser ;-)
> The textile support is expensive in regards to the memory allocated (freed
> afterwards).
> Details will come as soon as I find some time.

OK, no surprises there. For a single message there might even be more
than one text area which is parsed by textile, as our own MsgParser
must process the message first into text, hashtags, users and links.
As you mentioned, the objects from the parse tree are short-lived
though so I don't expect them to even leave the eden space on the Sun
VM. Not sure how big a performance impact this will have.

Re: Performance Tests - Suggestions and Ideas

[Markus Kohler <ma...@gmail.com>]

Hi all,
Just as a teaser ;-)
The textile support is expensive in regards to the memory allocated (freed
afterwards).
Details will come as soon as I find some time.

Regards,
Markus

On Fri, Nov 20, 2009 at 1:58 PM, Vassil Dichev <vd...@apache.org> wrote:

> I just wanted to say thanks to Markus for doing this, and I think
> performance testing is a critical aspect for this type of application
> (as Twitter have demonstrated very convincingly).
>
> I also believe performance testing of ESME is also a great way to test
> Lift, as ESME is one of the biggest open-source projects using Lift
> (ok, there aren't many anyway). Other bugs which performance tests
> could catch would be deadlocks and thread contention, and these are
> vitally important in the light of the dynamic change Lift is
> undergoing swapping its actor implementations.
>
> It's also a good idea to run performance tests on Hudson, at least for
> the time being- we don't have that many commits per week yet.
>
> About memory analysis being a low-hanging fruit: agree 100%, provided
> one condition is fulfilled. The condition is that we're able to test
> the application locally, and performance tests help us do exactly
> that. Previously many bugs could only be reproduced by everyone on the
> Stax instance. Besides, a long time ago I have worked on memory leak
> problems in the same area as Markus, so I think we have enough
> know-how.
>
> Vassil
>

Re: Performance Tests - Suggestions and Ideas

[Vassil Dichev <vd...@apache.org>]

I just wanted to say thanks to Markus for doing this, and I think
performance testing is a critical aspect for this type of application
(as Twitter have demonstrated very convincingly).

I also believe performance testing of ESME is also a great way to test
Lift, as ESME is one of the biggest open-source projects using Lift
(ok, there aren't many anyway). Other bugs which performance tests
could catch would be deadlocks and thread contention, and these are
vitally important in the light of the dynamic change Lift is
undergoing swapping its actor implementations.

It's also a good idea to run performance tests on Hudson, at least for
the time being- we don't have that many commits per week yet.

About memory analysis being a low-hanging fruit: agree 100%, provided
one condition is fulfilled. The condition is that we're able to test
the application locally, and performance tests help us do exactly
that. Previously many bugs could only be reproduced by everyone on the
Stax instance. Besides, a long time ago I have worked on memory leak
problems in the same area as Markus, so I think we have enough
know-how.

Vassil

Re: Performance Tests - Suggestions and Ideas

[Richard Hirsch <hi...@gmail.com>]

Just blogged about performance tests:
http://blog.esme.us/return-of-the-performance-test/

D.

On Thu, Nov 19, 2009 at 9:22 AM, Richard Hirsch <hi...@gmail.com> wrote:
> See my comments below.
>
> On Thu, Nov 19, 2009 at 9:12 AM, Markus Kohler <ma...@gmail.com> wrote:
>> Hi all,
>> See my comments below...
>>
>> On Thu, Nov 19, 2009 at 8:27 AM, Richard Hirsch <hi...@gmail.com>wrote:
>>
>>> I enhanced the performance test page in the wiki
>>> (http://cwiki.apache.org/confluence/display/ESME/Performance+tests)
>>> with more details and test ideas
>>>
>>>
>> I like that.
>>
>>
>>> @Markus: I think it would great if you could do the performance tests
>>> once a week on the Stax performance instance.  Initially, it would be
>>> great just to have test suite that we can run. It is critical that we
>>> can put such tests into our continuous integration cycle but until
>>> this is possible, it would be great if you could just do these short
>>> tests on a weekly basis. That would allow us to make sure that code
>>> changes aren't impacting our performance. I'd like to make these tests
>>> as easy for you to perform as possible.
>>>
>>>
>> The tests could also be used as simple functional tests. They would have
>> catched for example the javascript problem that we had lately.
>>
>>
>>> Although regression testing would be one goal, the changing of small
>>> parameters (adding one action for each user) and seeing the impact on
>>> performance would also be very interesting.
>>>
>>> Yes. I certainly need some simple extensions to what I've done so far. For
>> example users should follow other users.
>> I think this way we could even manually do some simple performance
>> tests/analysis.
>>
>>
>>> We should probably define some sort of a process that describes the
>>> tests to make sure that the basis is always the same. For example,
>>>
>>> 1. Drop existing DB - this can be via a stax command
>>>
>>
>> Yes for the regression test that is a must.
>>
>>
>>> 2. Create existing DB  - this can be via a stax command
>>>
>> Not sure, what you mean. Can we switch from one DB to another, for example
>> with different data sets (nr. of Users)?
>> That would be fantastic.
>
> You can only do this at deplyoment time. By changing some stax
> configuration files. No problem but only at deployment and
> not-run-time
>
>>
>>
>>> 3. Deploy application - - this can be via a stax command
>>> 4. Test - this could be done via maven
>>> 5. Take screenshots of Stax console (still have to define which
>>> screens we need - maybe like the ones you sent me last time)
>>>
>>
>> Stax doesn't have a way to get the data as a file (CSV or whatever)?
>
>
> You can access the mysql database via jdbc - there are details on the
> stax console page. This is probably the best way to do it. I didn't
> have access to these ports, because of corporate firewall
> restrictions.
>
>> I could use Selenium to navigate to the Stax console at the end of the test
>> and do some screenshots automatically.
>
> That would be very cool.
>
>>
>>
>>> 6. Mail all the details to me and I post to wiki
>>>
>>>
>> Yes. I can do that.
>>
>> My plan is to get the low hanging fruits first of course.
>> Those are (at the top of my head):
>> - Checking the client side. E.g. cache settings, request compression etc. .
>> Not sure whether I should do it now, because we will get a new UI hopefully
>> soon?
>> - memory usage analysis. This is low hanging, because I've done it so often,
>> I could do it while sleeping ;)
>> - quick memory allocation tracing analysis. E.g. how much memory is
>> allocated and reclaimed for one interaction step. Can be done manually, btw
>> does someone know how to tell maven "jetty:run" which JVM to use?
>>
>
> Added these ideas to the wiki page
>
>> Regards,
>> Markus
>>
>>
>>> Thoughts?
>>>
>>> D.
>>>
>>
>

Re: Performance Tests - Suggestions and Ideas

[Richard Hirsch <hi...@gmail.com>]

See my comments below.

On Thu, Nov 19, 2009 at 9:12 AM, Markus Kohler <ma...@gmail.com> wrote:
> Hi all,
> See my comments below...
>
> On Thu, Nov 19, 2009 at 8:27 AM, Richard Hirsch <hi...@gmail.com>wrote:
>
>> I enhanced the performance test page in the wiki
>> (http://cwiki.apache.org/confluence/display/ESME/Performance+tests)
>> with more details and test ideas
>>
>>
> I like that.
>
>
>> @Markus: I think it would great if you could do the performance tests
>> once a week on the Stax performance instance.  Initially, it would be
>> great just to have test suite that we can run. It is critical that we
>> can put such tests into our continuous integration cycle but until
>> this is possible, it would be great if you could just do these short
>> tests on a weekly basis. That would allow us to make sure that code
>> changes aren't impacting our performance. I'd like to make these tests
>> as easy for you to perform as possible.
>>
>>
> The tests could also be used as simple functional tests. They would have
> catched for example the javascript problem that we had lately.
>
>
>> Although regression testing would be one goal, the changing of small
>> parameters (adding one action for each user) and seeing the impact on
>> performance would also be very interesting.
>>
>> Yes. I certainly need some simple extensions to what I've done so far. For
> example users should follow other users.
> I think this way we could even manually do some simple performance
> tests/analysis.
>
>
>> We should probably define some sort of a process that describes the
>> tests to make sure that the basis is always the same. For example,
>>
>> 1. Drop existing DB - this can be via a stax command
>>
>
> Yes for the regression test that is a must.
>
>
>> 2. Create existing DB  - this can be via a stax command
>>
> Not sure, what you mean. Can we switch from one DB to another, for example
> with different data sets (nr. of Users)?
> That would be fantastic.

You can only do this at deplyoment time. By changing some stax
configuration files. No problem but only at deployment and
not-run-time

>
>
>> 3. Deploy application - - this can be via a stax command
>> 4. Test - this could be done via maven
>> 5. Take screenshots of Stax console (still have to define which
>> screens we need - maybe like the ones you sent me last time)
>>
>
> Stax doesn't have a way to get the data as a file (CSV or whatever)?


You can access the mysql database via jdbc - there are details on the
stax console page. This is probably the best way to do it. I didn't
have access to these ports, because of corporate firewall
restrictions.

> I could use Selenium to navigate to the Stax console at the end of the test
> and do some screenshots automatically.

That would be very cool.

>
>
>> 6. Mail all the details to me and I post to wiki
>>
>>
> Yes. I can do that.
>
> My plan is to get the low hanging fruits first of course.
> Those are (at the top of my head):
> - Checking the client side. E.g. cache settings, request compression etc. .
> Not sure whether I should do it now, because we will get a new UI hopefully
> soon?
> - memory usage analysis. This is low hanging, because I've done it so often,
> I could do it while sleeping ;)
> - quick memory allocation tracing analysis. E.g. how much memory is
> allocated and reclaimed for one interaction step. Can be done manually, btw
> does someone know how to tell maven "jetty:run" which JVM to use?
>

Added these ideas to the wiki page

> Regards,
> Markus
>
>
>> Thoughts?
>>
>> D.
>>
>

Re: Performance Tests - Suggestions and Ideas

[Markus Kohler <ma...@gmail.com>]

Hi all,
See my comments below...

On Thu, Nov 19, 2009 at 8:27 AM, Richard Hirsch <hi...@gmail.com>wrote:

> I enhanced the performance test page in the wiki
> (http://cwiki.apache.org/confluence/display/ESME/Performance+tests)
> with more details and test ideas
>
>
I like that.


> @Markus: I think it would great if you could do the performance tests
> once a week on the Stax performance instance.  Initially, it would be
> great just to have test suite that we can run. It is critical that we
> can put such tests into our continuous integration cycle but until
> this is possible, it would be great if you could just do these short
> tests on a weekly basis. That would allow us to make sure that code
> changes aren't impacting our performance. I'd like to make these tests
> as easy for you to perform as possible.
>
>
The tests could also be used as simple functional tests. They would have
catched for example the javascript problem that we had lately.


> Although regression testing would be one goal, the changing of small
> parameters (adding one action for each user) and seeing the impact on
> performance would also be very interesting.
>
> Yes. I certainly need some simple extensions to what I've done so far. For
example users should follow other users.
I think this way we could even manually do some simple performance
tests/analysis.


> We should probably define some sort of a process that describes the
> tests to make sure that the basis is always the same. For example,
>
> 1. Drop existing DB - this can be via a stax command
>

Yes for the regression test that is a must.


> 2. Create existing DB  - this can be via a stax command
>

Not sure, what you mean. Can we switch from one DB to another, for example
with different data sets (nr. of Users)?
That would be fantastic.


> 3. Deploy application - - this can be via a stax command
> 4. Test - this could be done via maven
> 5. Take screenshots of Stax console (still have to define which
> screens we need - maybe like the ones you sent me last time)
>

Stax doesn't have a way to get the data as a file (CSV or whatever)?
I could use Selenium to navigate to the Stax console at the end of the test
and do some screenshots automatically.


> 6. Mail all the details to me and I post to wiki
>
>
Yes. I can do that.

My plan is to get the low hanging fruits first of course.
Those are (at the top of my head):
- Checking the client side. E.g. cache settings, request compression etc. .
Not sure whether I should do it now, because we will get a new UI hopefully
soon?
- memory usage analysis. This is low hanging, because I've done it so often,
I could do it while sleeping ;)
- quick memory allocation tracing analysis. E.g. how much memory is
allocated and reclaimed for one interaction step. Can be done manually, btw
does someone know how to tell maven "jetty:run" which JVM to use?

Regards,
Markus


> Thoughts?
>
> D.
>