RE: [axis2] SimpleHTTPServer as transport listener causes too many connections to stay in TIME_WAIT

[Pär Malmqvist <pa...@hotmail.com>]

Hi Michele!
 
To catch the bug(s) I think the only way for you is to create a testcase that is as easy as possible for anyone to reproduce the bug without much effort in there own environment.
Most people writing here are good programmers but unfortenately they dont have that much time.I think you are doing a great job finding these issues with Axis2 and surrounding applications!
 
/Pär 
 
 
> Subject: Re: [axis2] SimpleHTTPServer as transport listener causes too many connections to stay in TIME_WAIT state> From: Michele.Mazzucco@ncl.ac.uk> To: axis-user@ws.apache.org> Date: Sat, 2 Feb 2008 20:31:51 +0000> > Paul,> > the NIO sender (used by OperationClient objects on the server side)> throws an ISE after a few minutes. > I have a logger on the OUT_FLOW phase which captured the error. Here is> the SOAP message:> > ERROR 20:21:08,751 (LoggingHandler.java:161) - OUT_FAULT_FLOW:> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope> xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"> xmlns:wsa="http://www.w3.org/2005/08/addressing"><soapenv:Header><wsa:To>http://sealbook.ncl.ac.uk:11001/axis2/services/anonService16</wsa:To><wsa:ReplyTo><wsa:Address>http://www.w3.org/2005/08/addressing/none</wsa:Address></wsa:ReplyTo><wsa:MessageID>urn:uuid:50C54185E1E92FA9931201983671032</wsa:MessageID><wsa:Action>http://www.w3.org/2005/08/addressing/soap/fault</wsa:Action><wsa:RelatesTo>urn:uuid:FFB56CB70BDEA2D6F41201983736482</wsa:RelatesTo></soapenv:Header><soapenv:Body><soapenv:Fault><soapenv:Code><soapenv:Value>soapenv:Receiver</soapenv:Value></soapenv:Code><soapenv:Reason><soapenv:Text xml:lang="en-US">Error while trying to forward the request to [http://giga23.ncl.ac.uk:18080/axis2/services/Service2] </soapenv:Text></soapenv:Reason><soapenv:Detail><Exception>org.apache.axis2.AxisFault: Error while trying to forward the request to > > > ..... hostname omitted ....> > at org.apache.axis2.AxisFault.makeFault(AxisFault.java:417)> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:406)> at> org.apache.axis2.receivers.RawXMLINOnlyMessageReceiver.invokeBusinessLogic(RawXMLINOnlyMessageReceiver.java:94)> at> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:96)> at> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)> at> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)> at> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120)> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)> at> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)> at> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)> at> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)> at> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)> at> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)> at> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)> at> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)> at> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)> at> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:887)> at org.apache.coyote.http11.Http11NioProtocol> $Http11ConnectionHandler.process(Http11NioProtocol.java:696)> at org.apache.tomcat.util.net.NioEndpoint> $SocketProcessor.run(NioEndpoint.java:2009)> at java.util.concurrent.ThreadPoolExecutor> $Worker.runTask(ThreadPoolExecutor.java:885)> at java.util.concurrent.ThreadPoolExecutor> $Worker.run(ThreadPoolExecutor.java:907)> at java.lang.Thread.run(Thread.java:619)> Caused by: java.lang.IllegalStateException: Error while trying to> forward the request to> [http://giga23.ncl.ac.uk:18080/axis2/services/Service2]> at> ncl.qosp.controller.service.streams.Sender.send(Sender.java:424)> at> ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.send(StreamServiceHandlerImpl.java:240)> at> ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.multipleForwards(StreamServiceHandlerImpl.java:226)> at> ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.releaseOrTryForward(StreamServiceHandlerImpl.java:505)> at> ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.response(StreamServiceHandlerImpl.java:581)> at> ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.response(StreamServiceHandlerImpl.java:46)> at> ncl.qosp.controller.dispatcher.streams.StreamDispatcherImpl.response(StreamDispatcherImpl.java:572)> at> ncl.qosp.controller.streams.StreamRoutingService.result(StreamRoutingService.java:705)> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)> at> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)> at java.lang.reflect.Method.invoke(Method.java:597)> at> org.apache.axis2.receivers.RawXMLINOnlyMessageReceiver.invokeBusinessLogic(RawXMLINOnlyMessageReceiver.java:90)> ... 20 more> Caused by: java.lang.IllegalStateException: I/O reactor has been shut> down> at> org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:176)> at> org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncRequest(HttpCoreNIOSender.java:295)> at> org.apache.synapse.transport.nhttp.HttpCoreNIOSender.invoke(HttpCoreNIOSender.java:218)> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:396)> at> org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:265)> at> org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)> at> ncl.qosp.controller.service.streams.Sender.send(Sender.java:399)> ... 31 more> > > The target node is up and it doesn't log any error, while Sender:399 is> the statement> OperationClient.execute(true);> > > To summarize about the new NIO components from synapse 1.1.1> - the NIO listener looks ok> - the NIO sender has still some problems> > > Michele> > > On Fri, 2008-02-01 at 19:59 +0000, Michele Mazzucco wrote:> > Paul,> > > > I did the following:> > - updated tomcat from 5.5 to 6.0.14> > - enabled and tuned the tomcat NIO connector> > - upgraded the NIO listener and sender to synapse 1.1.1 (I had to > > upgrade the http-core library from alpha5 to beta1)> > - enabled the NIO sender on the server side (as I said I have some > > ServiceClient instances running on the server too)> > > > but it didn't fix the problem. Indeed, about 50% of the connections > > in TIME_WAIT state are due to the tomcat connector and the other 50% > > to the axis2 sender.> > > > So I enabled the NIO sender and NIO receiver on the client side.> > My scenario is composed by about 80% of asynchronous interactions and > > 20% of very fast blocking in-out MEPs. Now the latter fail this way:> > > > ERROR 19:44:03,967 (QueueController.java:192) - Unable to add stream > > for queue Service2> > [java] org.apache.axis2.AxisFault: The input stream for an > > incoming message is null.> > [java] at > > org.apache.axis2.transport.TransportUtils.createSOAPMessage > > (TransportUtils.java:71)> > [java] at > > org.apache.axis2.description.OutInAxisOperationClient.handleResponse > > (OutInAxisOperation.java:326)> > [java] at > > org.apache.axis2.description.OutInAxisOperationClient.send > > (OutInAxisOperation.java:389)> > [java] at > > org.apache.axis2.description.OutInAxisOperationClient.executeImpl > > (OutInAxisOperation.java:211)> > [java] at org.apache.axis2.client.OperationClient.execute > > (OperationClient.java:163)> > [java] at org.apache.axis2.client.ServiceClient.sendReceive > > (ServiceClient.java:528)> > [java] at org.apache.axis2.client.ServiceClient.sendReceive > > (ServiceClient.java:508)> > [java] at > > ncl.qosp.test.client.streams.delayed.AddStream.send(AddStream.java:86)> > [java] at > > ncl.qosp.test.client.streams.delayed.QueueController.addStream > > (QueueController.java:163)> > [java] at > > ncl.qosp.test.client.streams.delayed.QueueController.init > > (QueueController.java:250)> > [java] at > > ncl.qosp.test.client.streams.delayed.QueueController.run > > (QueueController.java:264)> > [java] at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:650)> > [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run > > (ThreadPoolExecutor.java:675)> > [java] at java.lang.Thread.run(Thread.java:613)> > > > The executed statements (related to axis2) are:> > final OMElement payload = createPayload();> > > > ServiceClient sender = new ServiceClient > > (AddStreamConfiguration.getConfigurationContext(), null);> > sender.setOptions(getOptions());> > OMElement result = sender.sendReceive(payload);> > > > where the options are> > options.setTransportInProtocol(Constants.TRANSPORT_HTTP);> > options.setCallTransportCleanup(true);> > options.setTo(StreamClient.getRouterEpr());> > options.setAction(TestConstants.ADD_STREAM_SOAP_ACTION);> > options.setSoapVersionURI(SOAP12Constants.SOAP_ENVELOPE_NAMESPACE_URI);> > options.setProperty(HTTPConstants.CHUNKED, Boolean.TRUE);> > options.setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, > > Boolean.TRUE);> > > > and the configuration context differs from the default one because of > > the http connection manager with cached http client.> > > > > > Michele> > > > On 1 Feb 2008, at 16:35, Paul Fremantle wrote:> > > > > Michele> > >> > > You shouldn't need to change anything apart from the axis2 xml sender> > > and receiver sections.> > >> > > Paul> > >> > > On Feb 1, 2008 2:55 PM, Michele Mazzucco > > > <Mi...@ncl.ac.uk> wrote:> > >>> > >> On 1 Feb 2008, at 14:42, Paul Fremantle wrote:> > >>> > >>> Michele> > >>>> > >>> If you can post some test cases, we will do our best to improve the> > >>> NIO transport.> > >>> > >> Paul, as I mentioned I'll do it as soon as I find some time.> > >>> > >>> You also should try the NIO transport code from the> > >>> Synapse 1.1.1 release if you haven't already because that had a > > >>> number> > >>> of fixes in it.> > >>> > >> Thanks for the advice. Do I need to change anything apart from the> > >> nio section in axis2.xml (the HttpCoreNIOListener and> > >> HttpCoreNIOSender will be from package> > >> org.apache.synapse.transport.nhttp and not> > >> org.apache.axis2.transport.nhttp)?> > >>> > >> Thanks,> > >> Michele> > >>> > >>>> > >>> Thanks> > >>> Paul> > >>>> > >>> On Feb 1, 2008 1:35 PM, Michele Mazzucco> > >>> <Mi...@ncl.ac.uk> wrote:> > >>>>> > >>>> On 1 Feb 2008, at 03:15, robert lazarski wrote:> > >>>>> > >>>>> Me == clueless . Just some general advice though. Have you thought> > >>>>> about submitting patches to fix the NIO issues? I notice that> > >>>>> someone> > >>>>> from the synapse team (IIRC), three weeks ago, asked you to > > >>>>> submit a> > >>>>> test case for your failure in one of the jira's you posted.> > >>>>> > >>>> Robert, I know. I'll do that as soon as I find some time.> > >>>>>> > >>>>> As I've said before Michele, you easily have some of the most> > >>>>> complex> > >>>>> and long going use cases for axis2 as a non-committer. Your> > >>>>> involvement via patches certainly have a better chance of getting> > >>>>> developer attention.> > >>>>>> > >>>>> In respect to this issue, I can at least try to be a bit helpful:> > >>>>>> > >>>>> 1) Have you tried running your code on anything else besides > > >>>>> OSX to> > >>>>> see if these socket issues are OS related? I've run into several> > >>>>> socket and nio issues specific to linux for example.> > >>>>> > >>>> Yes. Indeed my problem occurs on a 4 CPUs linux box> > >>>>> > >>>>>> > >>>>> 2) Have you tried running http 1.0 instead of 1.1 ?> > >>>>> > >>>> Yes, it doesn't make any difference.> > >>>>>> > >>>>> 3) SimpleHTTPServer has never been meant for production use, so > > >>>>> stop> > >>>>> trying to use it like that. Fixing the nio issues seem to me to> > >>>>> be the> > >>>>> better path.> > >>>>>> > >>>> Yes, but the nio code comes from Synapse... and this would mean> > >>>> digging into the Synapse code (which I don't know).> > >>>>> > >>>>> My understanding of TIME_WAIT, via an old usenet post of mine, was> > >>>>> best explained to me this way:> > >>>>>> > >>>>> "After the connection is closed, there might still be some stray> > >>>>> packets that were delayed and could still arrive. The TIME_WAIT> > >>>>> status retains a record of a recent connection, so that the system> > >>>>> can recognize these as delay packets."> > >>>>>> > >>>>> Are these connections going from CLOSE_WAIT to TIME_WAIT? Can> > >>>>> you do> > >>>>> a 'netstat -anp' and show the transition states?> > >>>>> > >>>> Well, netstat doesn't show me any transition, only the current > > >>>> state> > >>>>> > >>>>> What issue do you> > >>>>> have with TIME_WAIT exactly?> > >>>>> > >>>> The number of connections in TIME_WAIT state grows very fast to a> > >>>> value > 500, and after that the my app. hangs (without crashing, it> > >>>> just freezes)> > >>>>>> > >>>>> Anyways, since SimpleHTTPServer is deep involved into the several> > >>>>> hundred unit / integration tests, I don't see major changes> > >>>>> happening> > >>>>> at this point.> > >>>>>> > >>>>> 4) Obviously try the latest axis2 nightlies and post questions to> > >>>>> the> > >>>>> http commons / reactor list. Its possible the latest snapshot of> > >>>>> reactor fixes your issue and can be promoted to the upcomming > > >>>>> axis2> > >>>>> 1.4 .> > >>>>> > >>>> I'll try.> > >>>>>> > >>>>> HTH,> > >>>>> Robert> > >>>>>> > >>>>> > >>>> Michele> > >>>>> > >>>>> > >>>>> On Jan 31, 2008 7:48 PM, Michele Mazzucco> > >>>>> <Mi...@ncl.ac.uk> wrote:> > >>>>>> Hello again,> > >>>>>>> > >>>>>> I've tried to write a TransportListener which uses Jetty as http> > >>>>>> server> > >>>>>> -- but I had no success: when requests come I get the following> > >>>>>> error> > >>>>>>> > >>>>>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope> > >>>>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"> > >>>>>> xmlns:wsa="http://www.w3.org/2005/08/> > >>>>>> addressing"><soapenv:Header><wsa:Action>http://www.w3.org/ > > >>>>>> 2005/08/> > >>>>>> addressing/soap/fault</wsa:Action></> > >>>>>> soapenv:Header><soapenv:Body><soapenv:Fault><faultcode>soapenv:Se > > >>>>>> rv> > >>>>>> er> > >>>>>> </faultcode><faultstring>java.lang.NullPointerException</> > >>>>>> faultstring><detail><Exception>org.apache.axis2.AxisFault:> > >>>>>> java.lang.NullPointerException> > >>>>>> at> > >>>>>> org.apache.axis2.transport.http.AxisServlet.doPost> > >>>>>> (AxisServlet.java:182)> > >>>>>> at javax.servlet.http.HttpServlet.service > > >>>>>> (HttpServlet.java:> > >>>>>> 709)> > >>>>>> at javax.servlet.http.HttpServlet.service > > >>>>>> (HttpServlet.java:> > >>>>>> 802)> > >>>>>> at> > >>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter> > >>>>>> (ApplicationFilterChain.java:252)> > >>>>>> at> > >>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter> > >>>>>> (ApplicationFilterChain.java:173)> > >>>>>> at> > >>>>>> org.apache.catalina.core.StandardWrapperValve.invoke> > >>>>>> (StandardWrapperValve.java:213)> > >>>>>> at> > >>>>>> org.apache.catalina.core.StandardContextValve.invoke> > >>>>>> (StandardContextValve.java:178)> > >>>>>> at> > >>>>>> org.apache.catalina.core.StandardHostValve.invoke> > >>>>>> (StandardHostValve.java:126)> > >>>>>> at> > >>>>>> org.apache.catalina.valves.ErrorReportValve.invoke> > >>>>>> (ErrorReportValve.java:105)> > >>>>>> at> > >>>>>> org.apache.catalina.core.StandardEngineValve.invoke> > >>>>>> (StandardEngineValve.java:107)> > >>>>>> at> > >>>>>> org.apache.catalina.connector.CoyoteAdapter.service> > >>>>>> (CoyoteAdapter.java:148)> > >>>>>> at> > >>>>>> org.apache.coyote.http11.Http11Processor.process> > >>>>>> (Http11Processor.java:869)> > >>>>>> at org.apache.coyote.http11.Http11BaseProtocol> > >>>>>> $Http11ConnectionHandler.processConnection > > >>>>>> (Http11BaseProtocol.java:> > >>>>>> 664)> > >>>>>> at> > >>>>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket> > >>>>>> (PoolTcpEndpoint.java:527)> > >>>>>> at> > >>>>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt> > >>>>>> (LeaderFollowerWorkerThread.java:80)> > >>>>>> at org.apache.tomcat.util.threads.ThreadPool> > >>>>>> $ControlRunnable.run(ThreadPool.java:684)> > >>>>>> at java.lang.Thread.run(Thread.java:595)> > >>>>>> Caused by: java.lang.NullPointerException> > >>>>>> at> > >>>>>> ncl.qosp.modules.manager.RouterDispatcher.isFault> > >>>>>> (RouterDispatcher.java:230)> > >>>>>> at> > >>>>>> ncl.qosp.modules.manager.RouterDispatcher.invoke> > >>>>>> (RouterDispatcher.java:267)> > >>>>>> at org.apache.axis2.engine.Phase.invoke(Phase.java:292)> > >>>>>> at> > >>>>>> org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:212)> > >>>>>> at> > >>>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:132)> > >>>>>> at> > >>>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPos > > >>>>>> tR> > >>>>>> eq> > >>>>>> uest(HTTPTransportUtils.java:275)> > >>>>>> at> > >>>>>> org.apache.axis2.transport.http.AxisServlet.doPost> > >>>>>> (AxisServlet.java:120)> > >>>>>> ... 16 more> > >>>>>>> > >>>>>> I'm attaching my implementation.> > >>>>>>> > >>>>>>> > >>>>>> Any help is appreciated,> > >>>>>> Michele> > >>>>>>> > >>>>>>> > >>>>>> On Thu, 2008-01-31 at 19:47 +0000, Michele Mazzucco wrote:> > >>>>>>> Hello everybody,> > >>>>>>>> > >>>>>>> I'm facing a serious problem with too many connections left in> > >>>>>>> TIME_WAIT> > >>>>>>> state, causing my system to hang.> > >>>>>>> Since on the server side I run some ServiceClient instances as> > >>>>>>> well I> > >>>>>>> wasn't sure about the root of my problem, but after some> > >>>>>>> investigations> > >>>>>>> I've found out that the responsible for this is the> > >>>>>>> SimpleHTTPServer> > >>>>>>> used as transport receiver for incoming http requests.> > >>>>>>>> > >>>>>>> Given that the NIO listener/sender are not stable (see the JIRAs> > >>>>>>> below)> > >>>>>>> I have to stick with these. Is there anything I can do?> > >>>>>>> Is there any way to inject a custom http connection manager and> > >>>>>>> manually> > >>>>>>> clean up the unused connections?> > >>>>>>>> > >>>>>>>> > >>>>>>> Thanks in advance,> > >>>>>>> Michele> > >>>>>>>> > >>>>>>>> > >>>>>>> [1] https://issues.apache.org/jira/browse/AXIS2-3473> > >>>>>>> [2] http://www.mail-archive.com/axis-user@ws.apache.org/> > >>>>>>> msg37314.html> > >>>>>>> [3] https://issues.apache.org/jira/browse/AXIS2-3428> > >>>>>>>> > >>>>>>>> > >>>>>>> ---------------------------------------------------------------- > > >>>>>>> --> > >>>>>>> --> > >>>>>>> -> > >>>>>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >>>>>>> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>>>>>>> > >>>>>>> > >>>>>> ----------------------------------------------------------------- > > >>>>>> --> > >>>>>> --> > >>>>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >>>>>> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>>>>>> > >>>>>> > >>>>> ------------------------------------------------------------------ > > >>>>> --> > >>>>> -> > >>>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >>>>> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>>>>> > >>>>> > >>>>> > >>>> ------------------------------------------------------------------- > > >>>> --> > >>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >>>> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>> --> > >>> Paul Fremantle> > >>> Co-Founder and VP of Technical Sales, WSO2> > >>> OASIS WS-RX TC Co-chair> > >>>> > >>> blog: http://pzf.fremantle.org> > >>> paul@wso2.com> > >>>> > >>> "Oxygenating the Web Service Platform", www.wso2.com> > >>>> > >>> -------------------------------------------------------------------- > > >>> -> > >>> > >>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >>> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>>> > >>> > >>> > >> ---------------------------------------------------------------------> > >> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > >> For additional commands, e-mail: axis-user-help@ws.apache.org> > >>> > >>> > >> > >> > >> > > -- > > > Paul Fremantle> > > Co-Founder and VP of Technical Sales, WSO2> > > OASIS WS-RX TC Co-chair> > >> > > blog: http://pzf.fremantle.org> > > paul@wso2.com> > >> > > "Oxygenating the Web Service Platform", www.wso2.com> > >> > > ---------------------------------------------------------------------> > > To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > > For additional commands, e-mail: axis-user-help@ws.apache.org> > >> > > > > > ---------------------------------------------------------------------> > To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> > For additional commands, e-mail: axis-user-help@ws.apache.org> > > > > ---------------------------------------------------------------------> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> For additional commands, e-mail: axis-user-help@ws.apache.org> 
_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/