Re: Time out problem when using Axis2 and Synapse together

[Vidura Gamini Abhaya <vi...@gmail.com>]

Thanks for the suggestions Amila. I will try to make these changes and
see whether it makes a difference. I remember trying out increasing
the listeners on Axis2 and trying it out once (it was sometime ago)
but I will do it properly and see whether this works. When I conducted
the experiments back then I had to change some parameters to get over
the problem (i.e. not to create that much of load on the server).
However, I'd like to find out whether Crishantha had any improvements
to his problem and what changes he made.

Thanks,

Vidura



On Sun, Dec 26, 2010 at 12:33 PM, Amila Suriarachchi
<am...@gmail.com> wrote:
>
>
> On Mon, Mar 1, 2010 at 11:01 AM, Vidura Gamini Abhaya <vi...@gmail.com>
> wrote:
>>
>> Hi All,
>>
>> I'm using Axis2 and Synapse together for some experiments. Axis2 ver 1.4
>> is used both as a client as well as to host a couple of web services. I use
>> Synapse (ver 1.2) as a load balancer/request redirector to 2 or more Axis2
>> instances. Axis2 servers run with a modified axis2-kernel where I've done
>> changes to AxisEngine and some of the transports. Synapse also uses this
>> modified kernel. I generate requests which gets load balanced according to
>> some algorithms I implement in Synapse. I change the arrival rates of the
>> requests (at Synapse) for various experiments.
>>
>> I encounter some problems when there's a higher arrival of requests at
>> synapse (for lower arrival rates this seem to work fine). After some time
>> into the experiments, some of the requests time out on the client side
>> complaining there's no response from Synapse. When inspecting Synapse, I
>> could see that the following exceptions were thrown,
>>
>> "[HttpClientWorker-4] ERROR ServerHandler IO error submiting response :
>> Connection is closed
>> org.apache.http.ConnectionClosedException: Connection is closed
>> at
>> org.apache.http.impl.nio.NHttpConnectionBase.assertNotClosed(NHttpConnectionBase.java:238)
>> at
>> org.apache.http.impl.nio.DefaultNHttpServerConnection.submitResponse(DefaultNHttpServerConnection.java:200)
>> at
>> org.apache.synapse.transport.nhttp.ServerHandler.commitResponse(ServerHandler.java:213)
>> at
>> org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncResponse(HttpCoreNIOSender.java:416)
>> at
>> org.apache.synapse.transport.nhttp.HttpCoreNIOSender.invoke(HttpCoreNIOSender.java:271)
>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:567)
>> at
>> org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:143)
>> at
>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:219)
>> at
>> org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:154)
>> at
>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:58)
>> at
>> org.apache.synapse.mediators.filters.OutMediator.mediate(OutMediator.java:60)
>> at
>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:58)
>> at
>> org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:125)
>> at
>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:176)
>> at
>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:344)
>> at
>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:139)
>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:274)
>> at
>> org.apache.synapse.transport.nhttp.ClientWorker.run(ClientWorker.java:200)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:713)"
>>
>> The number of requests synapse handles are logged and strangely that
>> equals to more than the number of requests sent to Synapse. Investigating
>> further on this I could see that Synapse had processed the same request more
>> than once and sometimes, the result hasn't been sent to the client. I'm
>> guessing that this may be due to retransmission of requests from the client
>> due to not having any response from synapse.
>>
>> The Axis2 client is used an asynchronous client and since the timing of
>> the requests is important, each request is created on a separate thread and
>> use a newly created serviceClient object inside each thread of execution.
>> After the non blocking call is done serviceClient.cleanup() and
>> serviceClient.cleanupTransport() is called. I'm wondering whether the
>> problems are connected with the version of the client I use (I've seen that
>> 1.4 has some problems) and/or the way I'm using the client. Here's a part of
>> the code how it's done,
>
> For synapse issue try to increase the time out value in nhttp properties
> file.
>>
>> "ServiceClient serviceClient = new ServiceClient();
>> Options opts = new Options();
>>
>> opts.setTo(EPR);
>> opts.setAction(Action);
>>
>> opts.setTimeOutInMilliSeconds(5400000);
>> serviceClient.setOptions(opts);
>>
>> serviceClient.sendReceiveNonBlocking(this.createPayload(parameter),
>> callback);
>> ...
>> serviceClient.cleanup();
>> serviceClient.cleanupTransport();"
>
> you can not clean up the transports before getting the response.
>
>>
>> Migrating the Axis2 client libraries to 1.5.1 resulted in exceptions being
>> thrown and requests not reaching Synapse at all. I have a couple of
>> questions on the above,
>
> With axis2 1.5.1 by default you have only 2 connections. Unless you increase
> this value it may gives exceptions for a high load scenarios.
>
> thanks,
> Amila.
>>
>> 1) Could this be due to bugs with the 1.4 version of Axis2?
>> 2) Are there any changes required/suggested in the way the client code
>> works?
>> 3) Could this be happening due to any problems with Synapse? (I posted
>> this to the Axis2 mailing list assuming that it might be to do with the
>> client), in which case please cross post this to the Synapse mailing list,
>> thanks.
>>
>> Appreciate any help and suggestions you could give in solving this issue.
>>
>> Thanks,
>>
>> Vidura
>>
>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@axis.apache.org
For additional commands, e-mail: java-user-help@axis.apache.org