[Axis2] Performance

[Dennis Sosnoski <dm...@sosnoski.com>]

I converted the program used for my web service performance tests last 
year (http://www.sosnoski.com/presents/cleansoap/comparing.html) to 
Axis2/XMLBeans 0.91. For comparison purposes I only used Axis 1.2.1 
doc/lit for now. In the initial version I just used the XMLBeans 
generated classes directly, like ordinary JavaBeans (the same approach 
I'd used with all the other tests). Here's how that came out in a run on 
my notebook (running the same series of four different request-response 
patterns as used in the earlier tests, where the number of queries goes 
down but the volume of response data goes up):

Total elapsed time for test 170981 ms. with 48 results
Total elapsed time for test 86784 ms. with 853 results
Total elapsed time for test 22623 ms. with 3052 results
Total elapsed time for test 16479 ms. with 6155 results

Axis 1.2.1, by comparison, gave these results:

Total elapsed time for test 5869 ms. with 48 results
Total elapsed time for test 6005 ms. with 853 results
Total elapsed time for test 7048 ms. with 3052 results
Total elapsed time for test 9999 ms. with 6155 results

Next I changed the code to use the XMLBeans objects only for reading and 
writing XML - all the comparison code on the server that handles the 
query I changed to use regular JavaBean-style copies of the data. Here's 
the results with that approach:

Total elapsed time for test 6237 ms. with 48 results
Total elapsed time for test 5053 ms. with 853 results
Total elapsed time for test 5744 ms. with 3052 results
Total elapsed time for test 7809 ms. with 6155 results

That's looking much more reasonable, and better than Axis 1.2.1 in most 
cases. From the pattern of the times it looks like Axis2 may have higher 
per-request overhead than Axis 1.2.1 at this point (the first test case, 
where Axis2 is slower, has 200 queries, the second only 100, etc.). I 
don't know why this would be the case, but it might be worth some simple 
tests to confirm or refute this.

The other known issue that definitely hurts Axis2/XMLBeans performance 
is that the output processing pulls the XML out of the XMLBeans 
framework and uses it to build an OM representation, which then gets 
serializer out. Once this is fixed so that XMLBeans writes directly to 
output (where possible - we can't do that if WS-Security is going to 
massage some of the data, or in other similar cases) the latter times in 
the above results should be considerably better.

I'm pretty snowed under until about the end of the month, but will try 
to get the test code added into the project if someone can suggest a 
place to put it. I don't speak Maven, but have Ant projects to build the 
code and .sh scripts for running the tests.

  - Dennis

Re: [Axis2] Performance

[Ajith Ranabahu <aj...@gmail.com>]

Hi Dennis,
Even if the results are not so entertaining we should thank you for doing 
this. Infact we have been trying to put up a jmeter performance test suite 
to compare the performance (and find possible performance bottlenecks)
It'll be great if you can post the code for this somewhere :)



On 9/15/05, Dennis Sosnoski <dm...@sosnoski.com> wrote:
> 
> I converted the program used for my web service performance tests last
> year (http://www.sosnoski.com/presents/cleansoap/comparing.html) to
> Axis2/XMLBeans 0.91. For comparison purposes I only used Axis 1.2.1
> doc/lit for now. In the initial version I just used the XMLBeans
> generated classes directly, like ordinary JavaBeans (the same approach
> I'd used with all the other tests). Here's how that came out in a run on
> my notebook (running the same series of four different request-response
> patterns as used in the earlier tests, where the number of queries goes
> down but the volume of response data goes up):
> 
> Total elapsed time for test 170981 ms. with 48 results
> Total elapsed time for test 86784 ms. with 853 results
> Total elapsed time for test 22623 ms. with 3052 results
> Total elapsed time for test 16479 ms. with 6155 results
> 
> Axis 1.2.1, by comparison, gave these results:
> 
> Total elapsed time for test 5869 ms. with 48 results
> Total elapsed time for test 6005 ms. with 853 results
> Total elapsed time for test 7048 ms. with 3052 results
> Total elapsed time for test 9999 ms. with 6155 results
> 
> Next I changed the code to use the XMLBeans objects only for reading and
> writing XML - all the comparison code on the server that handles the
> query I changed to use regular JavaBean-style copies of the data. Here's
> the results with that approach:
> 
> Total elapsed time for test 6237 ms. with 48 results
> Total elapsed time for test 5053 ms. with 853 results
> Total elapsed time for test 5744 ms. with 3052 results
> Total elapsed time for test 7809 ms. with 6155 results
> 
> That's looking much more reasonable, and better than Axis 1.2.1 in most
> cases. From the pattern of the times it looks like Axis2 may have higher
> per-request overhead than Axis 1.2.1 at this point (the first test case,
> where Axis2 is slower, has 200 queries, the second only 100, etc.). I
> don't know why this would be the case, but it might be worth some simple
> tests to confirm or refute this.
> 
> The other known issue that definitely hurts Axis2/XMLBeans performance
> is that the output processing pulls the XML out of the XMLBeans
> framework and uses it to build an OM representation, which then gets
> serializer out. Once this is fixed so that XMLBeans writes directly to
> output (where possible - we can't do that if WS-Security is going to
> massage some of the data, or in other similar cases) the latter times in
> the above results should be considerably better.
> 
> I'm pretty snowed under until about the end of the month, but will try
> to get the test code added into the project if someone can suggest a
> place to put it. I don't speak Maven, but have Ant projects to build the
> code and .sh scripts for running the tests.
> 
> - Dennis
> 



-- 
Ajith Ranabahu