Axis Performance

[David Turner <dt...@atl.lmco.com>]

Hello,

I have recently completed a series of simple performance measurements
for Apache Axis. I wanted to share the results and see if I could get
any new insights on the data.

To check out the results you should go to the following URL:

http://www.atl.external.lmco.com/projects/QoS/compare/dist_oo_compare_ipc.html

Here you will see a tool developed here at Lockheed Martin's Advanced
Technology Laboratory for comparing various forms of IPC. There are a
ton of results for all different types of middleware solutions. You
can browse through them by looking at the links in the left frame. If
you want to go directly to the various SOAP results please use the
filter located at the top of the left frame. You can enter the word
'xml', without the quotes, into the box and hit enter. All of our
recent results for web services should appear. It should look
something like this:

smp/
       web-service-and-xml-rpc/
            apache-axis/
                 v1.1_attachments
                 v1.1_dime
                 v1.1_inline
            apache-xml-rpc/
                 v1.2-b1
            XSOAP/
                 v1.2.23

First and foremost, if you click on the "apache-axis/" link you will
get our attempt at a detailed explanation of all the conditions in
which these tests were conducted. You can then look at the three sets
of results my clicking on either, SOAP with attachments, DIME, or
simple base64 encoded messages(byte arrays). All of our tests are
simple round trip latency tests that involve sending a message of x
bytes and waiting for an acknowledgment. We time this exchange for a
size of x varying from 4 to 65536 on every power of 2. The results you
are looking at should all be 1 million samples for each message size.

I have attached a graph I made real quick using the above tool. It
contains a comparison of apache-axis to a couple other
middleware's. These include basic tcp, apache implementation of
xml-rpc, XSOAP (aka soaprmi), and the TAO orb. We are trying to
understand a number of things.

Firstly, where is the large overhead coming from? Obviously the use
of XML in the SOAP protocol makes for more verbose messages. In
addition, we also considered the creation of a new connection for each
RPC(we are currently looking into developing some simple tests to
examine this issue). Are these two issues enough to explain this kind
of overhead or are we not considering something?

Secondly, if you look closely at the curves for XSOAP and apache-axis
base64 messages they are extremely similar. We do not usually see
results that are this close when testing two different implementations
of any middleware. Would anyone know of a specific reason why these
two implementations would be so close?

Finally, you can see that there is a break in performance of axis with
attachments and DIME at around 16384 size messages. We are assuming
this is because that is when axis begins writing temp files for every
attachment to the attachments/ directory. Is there a way to turn this
behavior off?

Thanks for reading this long email and I am anxious to here any
comments or questions.

David Turner
ATL