You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by "Aberbach, Joel" <Jo...@tfn.com> on 2001/01/12 16:08:22 UTC
Performance testing
I'm new to Cocoon and Tomcat and was interested in doing some performance
tests comparing Cocoon with a simple JSP page, the results of which I've
posted below. I apologize if anyone finds this long-winded. I would
appreciate any response that people have regarding the findings. Has anyone
found similar or different results from their own tests? Any comments or
suggestions are welcome.
Thanks,
Joel Aberbach
Test Results for Cocoon and JSP
Conditions
----------
Intel Pentium 3 500mhz
256M RAM
NT 4.0 SP 6
JDK 1.3
Tomcat 3.1 running standalone
Cocoon 1.8
50 hits to the server in each test
25 runs per test
Scenarios
----------
Scenario 1 - 1 Thread, default JVM heap size
Scenario 2 - 20 Threads, default JVM heap size
Scenario 3 - 1 Thread, maximum JVM heap size for test process = 64MB
Scenario 4 - 20 Threads, maximum JVM heap size for test process = 64MB
Scenario 5 - 1 Thread, maximum JVM heap size for Tomcat and test process =
64MB
Scenario 6 - 20 Threads, maximum JVM heap size for Tomcat and test process =
64MB
Measurements
------------
Memory Usage (MB) - the Tomcat/Cocoon memory usage statistic in megabytes
after running a particular test.
Max Time (sec) - the maximum time in seconds for a single run within a test.
For multi-thread tests this is
the average of each thread's maximum time.
Avg. Time (sec) - the average time in seconds over all the runs in a test
(25 in this case). For multi-thread
tests this is the average of each thread's average time.
Note: these times are not produced from the Cocoon profiler, but from our
own test program.
Distinctions
-------------
Cocoon tests consisted of a post to the server with a custom producer
specified. As a response to the
post, the producer created an object to return the desired XML. After
adding processing instructions, there
was one XSL transformation into HTML, which was returned in the response.
JSP tests were much simpler. The JSP ignored any parameters that were
posted to it, and directly used an
object to create XML directly. There were no XSL transformations in the JSP
tests.
Cocoon Results
---------------
Memory Usage (MB) Max Time (sec) Avg. Time
(sec)
Scenario 1 23.05 7.09 2.72
Scenario 2 61.95 134.34 106.13
Scenario 3 22.77 9.47 2.93
Scenario 4 62.33 124.85* 89.14*
Scenario 5 23.00 7.56 2.91
Scenario 6 48.62 117.71 99.22
*Scenario 4 yielded oddly fast results during the first test. I ran two
more tests later since the results
seemed so unlikely, and both took a good deal longer. I don't know what
could account for the initial run,
but I averaged the times for the three runs in the results above.
JSP Results
------------
Memory Usage (MB) Max Time (sec) Avg. Time
(sec)
Scenario 1 9.50 1.83 0.89
Scenario 2 31.34 23.79 14.94
Scenario 3 9.70 1.98 1.03
Scenario 4 31.93 24.40 15.72
Scenario 5 9.78 1.97 1.01
Scenario 6 29.71 23.06 15.51
Conclusion
----------
With only 1 thread running, Cocoon runs about 3X slower than a simpler JSP
page. With more threads
Cocoon runs between 6-7X slower. These results are encouraging since Cocoon
has a lot to offer in terms
of infrastructure and flexibility of presentation, and it does not run
orders of magnitude slower than a
simple JSP page.
Notes
-----
In some cases exceptions were detected while running tests:
1. Cocoon under scenario 2 experienced 5 ConnectExceptions
2. Cocoon under scenario 4 experienced 3 ConnectExceptions (in one out of 3
repeated tests)
3. JSP under scenario 4 experienced 16 BindExceptions
4. JSP under scenario 6 experienced 3 BindExceptions
In all cases the exceptions occurred in the same part of the test code, when
trying to get the OutputStream
used for posting data to Tomcat/Cocoon. There isn't much evidence to show
that these exceptions are harmful, or
meaningful for a particular scenario.
In multi-thread Cocoon tests (scenarios 2, 4, and 6), the Tomcat/Cocoon java
process continued to
periodically use large amounts of CPU cycles after the test was over. It
didn't grab 100% at all times, but
rather used 50-100% every few seconds. This did not stop until Tomcat was
shutdown, even if the process
was left running overnight.
Other tests
-----------
When trying to run 100 threads, Cocoon (or Tomcat) was not consistently up
to handling the load. A
100-thread test caused a shutdown due to out of memory exceptions once. On
another 100-thread test with the,
Tomcat did not shutdown per-se, but it never completed the tests. In these
large thread tests, several SAX
Exceptions occurred, related to the inability to parse a given XSL document.
Memory usage averaged
around 80MB in these 100-thread tests.
Re: Performance testing
Posted by Berin Loritsch <bl...@apache.org>.
Aberbach, Joel wrote:
> I'm new to Cocoon and Tomcat and was interested in doing some performance
>
> tests comparing Cocoon with a simple JSP page, the results of which I've
>
> posted below. I apologize if anyone finds this long-winded. I would
>
> appreciate any response that people have regarding the findings. Has anyone
>
> found similar or different results from their own tests? Any comments or
>
> suggestions are welcome.
This is very useful information, but I would like to see a comparison with
Cocoon 2 under the same environments. It would shed some light on whether
some of our pooling code helps with stability and response times. No caching
is available yet--but it would be compelling to see the difference.
>
>
>
> Thanks,
>
> Joel Aberbach
>
>
>
> Test Results for Cocoon and JSP
>
>
>
> Conditions
>
> ----------
>
> Intel Pentium 3 500mhz
>
> 256M RAM
>
> NT 4.0 SP 6
>
> JDK 1.3
>
> Tomcat 3.1 running standalone
>
> Cocoon 1.8
>
> 50 hits to the server in each test
>
> 25 runs per test
>
>
>
> Scenarios
>
> ----------
>
> Scenario 1 - 1 Thread, default JVM heap size
>
> Scenario 2 - 20 Threads, default JVM heap size
>
> Scenario 3 - 1 Thread, maximum JVM heap size for test process = 64MB
>
> Scenario 4 - 20 Threads, maximum JVM heap size for test process = 64MB
>
> Scenario 5 - 1 Thread, maximum JVM heap size for Tomcat and test process =
>
> 64MB
>
> Scenario 6 - 20 Threads, maximum JVM heap size for Tomcat and test process =
>
> 64MB
>
>
>
> Measurements
>
> ------------
>
> Memory Usage (MB) - the Tomcat/Cocoon memory usage statistic in megabytes
>
> after running a particular test.
>
> Max Time (sec) - the maximum time in seconds for a single run within a test.
>
> For multi-thread tests this is
>
> the average of each thread's maximum time.
>
> Avg. Time (sec) - the average time in seconds over all the runs in a test
>
> (25 in this case). For multi-thread
>
> tests this is the average of each thread's average time.
>
>
>
> Note: these times are not produced from the Cocoon profiler, but from our
>
> own test program.
>
>
>
>
>
> Distinctions
>
> -------------
>
> Cocoon tests consisted of a post to the server with a custom producer
>
> specified. As a response to the
>
> post, the producer created an object to return the desired XML. After
>
> adding processing instructions, there
>
> was one XSL transformation into HTML, which was returned in the response.
>
>
>
> JSP tests were much simpler. The JSP ignored any parameters that were
>
> posted to it, and directly used an
>
> object to create XML directly. There were no XSL transformations in the JSP
>
> tests.
>
>
>
>
>
> Cocoon Results
>
> ---------------
>
> Memory Usage (MB) Max Time (sec) Avg. Time
>
> (sec)
>
> Scenario 1 23.05 7.09 2.72
>
> Scenario 2 61.95 134.34 106.13
>
> Scenario 3 22.77 9.47 2.93
>
> Scenario 4 62.33 124.85* 89.14*
>
> Scenario 5 23.00 7.56 2.91
>
> Scenario 6 48.62 117.71 99.22
>
>
>
> *Scenario 4 yielded oddly fast results during the first test. I ran two
>
> more tests later since the results
>
> seemed so unlikely, and both took a good deal longer. I don't know what
>
> could account for the initial run,
>
> but I averaged the times for the three runs in the results above.
>
>
>
>
>
> JSP Results
>
> ------------
>
> Memory Usage (MB) Max Time (sec) Avg. Time
>
> (sec)
>
> Scenario 1 9.50 1.83 0.89
>
> Scenario 2 31.34 23.79 14.94
>
> Scenario 3 9.70 1.98 1.03
>
> Scenario 4 31.93 24.40 15.72
>
> Scenario 5 9.78 1.97 1.01
>
> Scenario 6 29.71 23.06 15.51
>
>
>
>
>
> Conclusion
>
> ----------
>
> With only 1 thread running, Cocoon runs about 3X slower than a simpler JSP
>
> page. With more threads
>
> Cocoon runs between 6-7X slower. These results are encouraging since Cocoon
>
> has a lot to offer in terms
>
> of infrastructure and flexibility of presentation, and it does not run
>
> orders of magnitude slower than a
>
> simple JSP page.
>
>
>
>
>
> Notes
>
> -----
>
> In some cases exceptions were detected while running tests:
>
> 1. Cocoon under scenario 2 experienced 5 ConnectExceptions
>
> 2. Cocoon under scenario 4 experienced 3 ConnectExceptions (in one out of 3
>
> repeated tests)
>
> 3. JSP under scenario 4 experienced 16 BindExceptions
>
> 4. JSP under scenario 6 experienced 3 BindExceptions
>
>
>
> In all cases the exceptions occurred in the same part of the test code, when
>
> trying to get the OutputStream
>
> used for posting data to Tomcat/Cocoon. There isn't much evidence to show
>
> that these exceptions are harmful, or
>
> meaningful for a particular scenario.
>
>
>
> In multi-thread Cocoon tests (scenarios 2, 4, and 6), the Tomcat/Cocoon java
>
> process continued to
>
> periodically use large amounts of CPU cycles after the test was over. It
>
> didn't grab 100% at all times, but
>
> rather used 50-100% every few seconds. This did not stop until Tomcat was
>
> shutdown, even if the process
>
> was left running overnight.
>
>
>
>
>
> Other tests
>
> -----------
>
> When trying to run 100 threads, Cocoon (or Tomcat) was not consistently up
>
> to handling the load. A
>
> 100-thread test caused a shutdown due to out of memory exceptions once. On
>
> another 100-thread test with the,
>
> Tomcat did not shutdown per-se, but it never completed the tests. In these
>
> large thread tests, several SAX
>
> Exceptions occurred, related to the inability to parse a given XSL document.
>
> Memory usage averaged
>
> around 80MB in these 100-thread tests.
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: cocoon-users-unsubscribe@xml.apache.org
>
> For additional commands, e-mail: cocoon-users-help@xml.apache.org
Re: Performance testing
Posted by Donald Ball <ba...@webslingerZ.com>.
On Fri, 12 Jan 2001, Aberbach, Joel wrote:
> I'm new to Cocoon and Tomcat and was interested in doing some performance
> tests comparing Cocoon with a simple JSP page, the results of which I've
> posted below. I apologize if anyone finds this long-winded. I would
> appreciate any response that people have regarding the findings. Has anyone
> found similar or different results from their own tests? Any comments or
> suggestions are welcome.
>
> Distinctions
> -------------
> Cocoon tests consisted of a post to the server with a custom producer
> specified. As a response to the
> post, the producer created an object to return the desired XML. After
> adding processing instructions, there
> was one XSL transformation into HTML, which was returned in the response.
interesting, but i wish you'd compared JSP v.s. XSP - both compile pages
into special objects and then repeatedly execute those objects. actually,
you'd want to compare JSP to an XSP with a hasChanged method to an XSP
with no hasChanged method.
i bet that XSP with a hasChanged method that returns true much of the time
will be faster than the JSP page.
- donald