Performance Test

[Gerhard Froehlich <g-...@gmx.de>]

Hi Team,
it seems to be a tradition to exec some
perfomance tests before the next release ;-).

I did one tonight and here the results.

Machine:
x86 W2K 256MB 1,2 GH Athlon

JVM
- resin 2.04 
-  -Xms20000000 -Xmx50000000

Cocoon:
- Log level: ERROR
- Using new Store components in the scratchpad

Tool:
- Mircrosoft <spit> Web Application Stress Tools
- 20 Threads, 500 ms random delay
- Test period 30min

Observations:
- no memory or cpu leaks 
- compared to august last year, cocoon seems
to be a lot of quicker and stable
- Number of hits: 19502
- Requests per Second: 12.17

People, we are on the right way!

Report is attached as txt file.

  Gerhard
 
----------------------------
I just found the last bug...
---------------------------- 

Re: Performance Test

[Berin Loritsch <bl...@apache.org>]

Gerhard Froehlich wrote:

> Hi Team,
> it seems to be a tradition to exec some
> perfomance tests before the next release ;-).
> 
> I did one tonight and here the results.
> 
> Machine:
> x86 W2K 256MB 1,2 GH Athlon
> 
> JVM
> - resin 2.04 
> -  -Xms20000000 -Xmx50000000
> 
> Cocoon:
> - Log level: ERROR
> - Using new Store components in the scratchpad
> 
> Tool:
> - Mircrosoft <spit> Web Application Stress Tools
> - 20 Threads, 500 ms random delay
> - Test period 30min
> 
> Observations:
> - no memory or cpu leaks 
> - compared to august last year, cocoon seems
> to be a lot of quicker and stable
> - Number of hits: 19502
> - Requests per Second: 12.17
> 
> People, we are on the right way!


I performed my own performance test with a slightly more conservative machine

(i.e. 750 MHz Athlon).

I noticed some trends that are important to point out.

I used Apache JMeter running on the same machine, so my results are tainted.
However, during the life of the test, the average request time went up (this
is an average of all the resources being tested.  The Jisp store is definitely
better--the list of items in the cache (I upped the limit to 200) comes out to
roughly 21.

While request time started out at a comfortable 250ms average, it went up to
over a second average by the end of the test--still looking as if it was going
to increase more.

After I turned of the graph view, and serialized everything to disk, the request/
response time was _much_ quicker.  The most expensive being moreover.xml.

After I discovered this issue, I reran the site with double the number of threads
you specified (40).  My results are tainted in that I was doing other things while
running the test, but since my CPU was only running between 30 and 70% the results
are only minorly longer than otherwise would be.

The biggest problem was the connection to moreover kept timing out.

I will post my results here (with 40 threads):

URL 
					Min	Max	Avg	Std. Dev	Samples
http://localhost/cocoon/documents/index.html 
  20	   912	   74	   99		594
http://localhost/cocoon/hello.html 
	  0	   901	   45	   95		651
http://localhost/cocoon/hello.svg 
	  0	   821	   42	   76		630
http://localhost/cocoon/hello.wrl 
	  0	   821	   39	   90		629
http://localhost/cocoon/i18n/simple.xsp 
	 20 
   1352	  120	  168		597
http://localhost/cocoon/moreover/moreover.xml 
   0	123808	11710	19054 
	623
http://localhost/cocoon/slashdot/slashdot.xml 
390 
144527 
  6463	 8457		649
http://localhost/cocoon/welcome 
		 10	   891	   45	   85		651
http://localhost/cocoon/xsp/simple 
	 20 
   1202	   98	  146		596

With the exception of the number of samples, all measurements are in milliseconds.

The aggregated average for this spread is 2059 ms.  That's an average of .49 requests/second.
However, if we factor out the resources that have yet another connection to the net (slashdot
and moreover), the aggregated average becomes 66 ms.  That's an average of 15 requests/second.
Again, factoring out everything but the "hello.*" and "welcome" pages (the cheapest pages),
the average becomes 43 ms--which is 23 requests/second.

This also lends some insight to the use of certain generators and transformers.  For instance,
the xsp system adds an average of 55 ms to the response time.  The i18n transformer adds an
aditional average of 22 ms to the response time.

This also lends some insight to the stream generators that look off-site for information.  While
it is a cool idea, we must cache the information, and configure the ergodic time-period manually.
For instance, Slashdot has published the information that the site is statically updated every
half hour, which means that the ergodic period for a response from that site should be 30 minutes.

-- 

"They that give up essential liberty to obtain a little temporary safety
  deserve neither liberty nor safety."
                 - Benjamin Franklin


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org