You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ode.apache.org by Lavanya Ramakrishnan <la...@cs.indiana.edu> on 2007/11/27 23:21:31 UTC
Problem with multiple concurrent clients
I am running a simple test with multiple concurrent clients that make
repeated calls to the engine and getting the NullPointerException below.
My configuration is
Tomcat 5.5.23
MySQL for backend
ODE 1.1 subversion branch (only change is the patch Jira ODE-211 applied)
Example: HelloWorld
I noticed some previous messages with the same error which seemed to
indicate a problem with the axis-kernel jar. I noticed that the 1.1 branch
already has the 1.3 jar which is supposed to have fixed the problem.
The problem seems to definitely get worse with increasing concurrent
clients. With 5 clients and 50 back to back calls a large number of them
fail.
Any suggestions will be helpful.
thanks!
Lavanya
ERROR - GeronimoLog.error(108) | Error processing response for MEX
{MyRoleMex#hqejbhcnphr2s4q4a7l7qd [Client hqejbhcnphr2s4q4a7l7qc] calling
{http://ode/bpel/unit-test.wsdl}HelloService.hello(...)}
java.lang.NullPointerException: null message!
at
org.apache.ode.bpel.engine.MessageImpl.<init>(MessageImpl.java:40)
at
org.apache.ode.bpel.engine.MessageExchangeImpl.getResponse(MessageExchangeImpl.java:97)
at org.apache.ode.axis2.ODEService.onResponse(ODEService.java:226)
at
org.apache.ode.axis2.ODEService.onAxisMessageExchange(ODEService.java:172)
at
org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(ODEMessageReceiver.java:69)
at
org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(ODEMessageReceiver.java:50)
at
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:96)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)
at
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275)
at
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:595)
Re: Problem with multiple concurrent clients
Posted by jbi joe <jo...@daggerpoint.net>.
How bout a property setting. A Much easier solution for
those of us facing issues where periods of
high activity lead to timeouts. I would have to suggest
a configurable value would at least allow for a temporary
band aid, in lieu of diving in and changing how the BPEL
in an already existing project is working.
Just my 2 cents... Im curious as to what ended up
being the solution here?
TIA
Lavanya Ramakrishnan wrote:
>
>> The message exchange timeout is not configurable but it's also 2mn, which
>> is
>> fairly conservative. At this point I'm wondering if timing out is not the
>> correct behavior. Maybe there's a use case that I'm not seeing but beside
>> pushing load testing at its max, I can't see when you'd want to wait for
>> more than 2mn for a non-reliable transport over a request/response
>> exchange.
>> Don't you think?
>
> In terms of semantics, I am not sure if at this point timing out is the
> correct behavior either in this situation. But I can see that as a
> possible approach for sure. I think some of it goes to where you want to
> put the fault tolerance for excessive delayed response times.
>
> I guess it is true that excessive load might be the only case when we need
> longer timeouts. I guess otherwise clients can also use async response as
> a way for longer response times or variable response times from individual
> services.
>
> My concern was largely that with slightly complex workflows I might start
> seeing this even with lesser concurrent clients but I can probably
> experiment to find out if this is true. And I can probably tweak the value
> in code directly for when I am doing experiments with larger number of
> clients for now.
>
> thanks!
> Lavanya
>
>
--
View this message in context: http://www.nabble.com/Problem-with-multiple-concurrent-clients-tf4884888.html#a14140278
Sent from the Apache Ode User mailing list archive at Nabble.com.
Re: Problem with multiple concurrent clients
Posted by jbi joe <jo...@daggerpoint.net>.
Much easier solution for those of us facing issues where periods of
high activity lead to timeouts. I would have to suggest
a configurable value would at least allow for a temporary
band aid, in lieu of diving in and changing how the BPEL
in an already existing project is working.
Just my 2 cents... Im curious as to what ended up
being the solution here?
TIA
Lavanya Ramakrishnan wrote:
>
>> The message exchange timeout is not configurable but it's also 2mn, which
>> is
>> fairly conservative. At this point I'm wondering if timing out is not the
>> correct behavior. Maybe there's a use case that I'm not seeing but beside
>> pushing load testing at its max, I can't see when you'd want to wait for
>> more than 2mn for a non-reliable transport over a request/response
>> exchange.
>> Don't you think?
>
> In terms of semantics, I am not sure if at this point timing out is the
> correct behavior either in this situation. But I can see that as a
> possible approach for sure. I think some of it goes to where you want to
> put the fault tolerance for excessive delayed response times.
>
> I guess it is true that excessive load might be the only case when we need
> longer timeouts. I guess otherwise clients can also use async response as
> a way for longer response times or variable response times from individual
> services.
>
> My concern was largely that with slightly complex workflows I might start
> seeing this even with lesser concurrent clients but I can probably
> experiment to find out if this is true. And I can probably tweak the value
> in code directly for when I am doing experiments with larger number of
> clients for now.
>
> thanks!
> Lavanya
>
>
--
View this message in context: http://www.nabble.com/Problem-with-multiple-concurrent-clients-tf4884888.html#a14140278
Sent from the Apache Ode User mailing list archive at Nabble.com.
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
> The message exchange timeout is not configurable but it's also 2mn, which is
> fairly conservative. At this point I'm wondering if timing out is not the
> correct behavior. Maybe there's a use case that I'm not seeing but beside
> pushing load testing at its max, I can't see when you'd want to wait for
> more than 2mn for a non-reliable transport over a request/response exchange.
> Don't you think?
In terms of semantics, I am not sure if at this point timing out is the
correct behavior either in this situation. But I can see that as a
possible approach for sure. I think some of it goes to where you want to
put the fault tolerance for excessive delayed response times.
I guess it is true that excessive load might be the only case when we need
longer timeouts. I guess otherwise clients can also use async response as
a way for longer response times or variable response times from individual
services.
My concern was largely that with slightly complex workflows I might start
seeing this even with lesser concurrent clients but I can probably
experiment to find out if this is true. And I can probably tweak the value
in code directly for when I am doing experiments with larger number of
clients for now.
thanks!
Lavanya
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
On Dec 1, 2007 11:48 AM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
wrote:
> This is the error I see in the logs:
>
>
> ERROR - GeronimoLog.error(104) | Timeout or execution error when waiting
> for response to MEX {MyRoleMex#hqejbhcnphr2sems2kt6p1 [Client
> hqejbhcnphr2sems2kt6p0] calling {http://ode/bpel/unit-test.w\<http://ode/bpel/unit-test.w%5C>
> sdl}HelloService.hello(...)} java.util.concurrent.TimeoutException:
> Message exchange
>
> org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl$ResponseFuture@1ec697e
> timed out when waiting for a response!
>
The message exchange timeout is not configurable but it's also 2mn, which is
fairly conservative. At this point I'm wondering if timing out is not the
correct behavior. Maybe there's a use case that I'm not seeing but beside
pushing load testing at its max, I can't see when you'd want to wait for
more than 2mn for a non-reliable transport over a request/response exchange.
Don't you think?
Cheers,
Matthieu
>
>
>
> On Fri, 30 Nov 2007, Matthieu Riou wrote:
>
> > On Nov 30, 2007 5:10 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> > wrote:
> >
> > > Thanks! This is looking really good now!
> > >
> > > When I have a large number of connections (>75) I am beginning to see
> some
> > > TimeoutExceptions from the reply response. Is there a parameter I can
> > > tweak for the timeout value?
> > >
> >
> > Do you know exactly what times out or have a stack trace? Could just be
> your
> > client connection.
> >
> > Cheers,
> > Matthieu
> >
> >
> > >
> > > thanks again!
> > > Lavanya
> > >
> > > On Fri, 30 Nov 2007, Matthieu Riou wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry for taking so long but I've been side-tracked a couple of
> times
> > > and
> > > > this wasn't trivial to debug either. So I've found a couple of
> problems:
> > > >
> > > > 1. A potential deadlock (that was happening once in a while) in our
> JPA
> > > > Process DAO. Basically each new instance creation resulted in an
> update
> > > on
> > > > the very same process record to increase some counter, which was
> pretty
> > > bad.
> > > >
> > > > 2. A much more tricky bug with OpenJPA committing the transaction
> once
> > > in a
> > > > while without us asking for it. This appeared to be somewhat linked
> to
> > > the
> > > > Geronimo transaction manager as I've only seen that happening with
> it.
> > > So
> > > > once a while, when OpenJPA was getting a new set of sequence numbers
> to
> > > > assign to primary keys, it forced a commit in the middle of our
> > > transaction.
> > > > Hence errors about not having a transaction anymore afterward. I've
> > > upgraded
> > > > to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly
> for
> > > me
> > > > know.
> > > >
> > > > I've done some tests on Tomcat/MySQL with 30 threads and it worked
> > > properly.
> > > > I think there's a lot we could do to improve performances by
> optimizing
> > > the
> > > > queries but I don't have much time for that right now. But at still
> it
> > > works
> > > > and it's not ridiculously slow with enough threads. So update your
> > > version
> > > > of the 1.1 branch and you should see an improvement.
> > > >
> > > > Cheers,
> > > > Matthieu
> > > >
> > > > On Nov 28, 2007 5:28 PM, Matthieu Riou <ma...@offthelip.org>
> wrote:
> > > >
> > > > > Okay I can reproduce the problem, it's on Derby but I think in
> that
> > > case
> > > > > it doesn't make much difference. Seems to be a big database
> > > deadlock... I'll
> > > > > keep you posted.
> > > > >
> > > > > Cheers,
> > > > > Matthieu
> > > > >
> > > > >
> > > > > On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <
> laramakr@cs.indiana.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Just to clarify, I am only trying to run 5 parallel clients but
> they
> > > do
> > > > > > make repeated helloWorld calls.
> > > > > >
> > > > > > I added these properties. But I am still getting the NPE. but in
> > > > > > addition I am seeing another exception which might be a clue?
> > > > > >
> > > > > > The entire log is at
> > > http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > > > >
> > > > > >
> > > > > > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > > > > > ERROR - GeronimoLog.error(108) | Method "run" in class
> > > > > > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected
> exception.
> > > > > > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > > > > > synchronizer.
> > > > > > at
> > > > > >
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer
> > > (
> > > > > > SimpleScheduler.java:206)
> > > > > > at
> > > > > >
> > > org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > > > > > MyRoleMessageExchangeImpl.java :215)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > > > > > MessageExchangeImpl.java:173)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > > > > > BpelRuntimeContextImpl.java:544)
> > > > > > at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > > > > > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown
> > > Source)
> > > > > > at
> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > > > DelegatingMethodAccessorImpl.java :25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:585)
> > > > > > at
> > > > > > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(
> JacobVPU.java
> > > :451)
> > > > > > at org.apache.ode.jacob.vpu.JacobVPU.execute(
> JacobVPU.java:139)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > > > > > BpelRuntimeContextImpl.java:838)
> > > > > > at
> > > > > >
> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > > > > > PartnerLinkMyRoleImpl.java :197)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(
> > > BpelProcess.java
> > > > > > :167)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(
> > > BpelProcess.java
> > > > > > :335)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > > > > > BpelEngineImpl.java:328)
> > > > > > at
> > > > > > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > > > > > BpelServerImpl.java:365)
> > > > > > at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > > SimpleScheduler.java:341)
> > > > > > at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > > SimpleScheduler.java:340)
> > > > > > at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > > > > > SimpleScheduler.java:179)
> > > > > > at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > > SimpleScheduler.java:339)
> > > > > > at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > > SimpleScheduler.java:336)
> > > > > > at
> > > > > > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java
> :269)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java
> :123)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > > > > > ThreadPoolExecutor.java:650)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > > > > > ThreadPoolExecutor.java:675)
> > > > > > at java.lang.Thread.run(Thread.java:595)
> > > > > > Caused by: java.lang.NullPointerException
> > > > > > at
> > > > > >
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > > > > > SimpleScheduler.java:194)
> > > > > > ... 25 more
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> > > > > >
> > > > > > > In internal mode you should adjust the pool size with
> something
> > > like:
> > > > > > >
> > > > > > > ode-axis2.db.pool.max=100
> > > > > > > ode-axis2.db.pool.min=40
> > > > > > >
> > > > > > > With 100 threads and the default pool size (10) you're *very*
> > > probably
> > > > > > > exhausting the number of connections.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Matthieu
> > > > > > >
> > > > > > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > > > > > laramakr@cs.indiana.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > Lavanya, are you sure you configured a large enough
> connection
> > > > > > pool to
> > > > > > > > > handle all your requests? If you start a large number of
> > > parrallel
> > > > > > > > requests
> > > > > > > > > they'll all try to get a DB connection and if there's not
> > > enough
> > > > > > of them
> > > > > > > > > available, things run bad. Let me know.
> > > > > > > >
> > > > > > > > I also suspected a pool exhaustion. So based on earlier
> posts I
> > > am
> > > > > > using
> > > > > > > > db.mode to be INTERNAL. How do I increase the db connection
> pool
> > > in
> > > > > > this
> > > > > > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > > > > > >
> > > > > > > > Here is what I have so far in my ode-axis2.properites.
> > > > > > > >
> > > > > > > > ode-axis2.db.mode=INTERNAL
> > > > > > > > ode-axis2.db.int.jdbcurl=***
> > > > > > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > > > > > ode-axis2.threads.pool.size=200
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
This is the error I see in the logs:
ERROR - GeronimoLog.error(104) | Timeout or execution error when waiting
for response to MEX {MyRoleMex#hqejbhcnphr2sems2kt6p1 [Client
hqejbhcnphr2sems2kt6p0] calling {http://ode/bpel/unit-test.w\
sdl}HelloService.hello(...)} java.util.concurrent.TimeoutException:
Message exchange
org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl$ResponseFuture@1ec697e
timed out when waiting for a response!
On Fri, 30 Nov 2007, Matthieu Riou wrote:
> On Nov 30, 2007 5:10 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> wrote:
>
> > Thanks! This is looking really good now!
> >
> > When I have a large number of connections (>75) I am beginning to see some
> > TimeoutExceptions from the reply response. Is there a parameter I can
> > tweak for the timeout value?
> >
>
> Do you know exactly what times out or have a stack trace? Could just be your
> client connection.
>
> Cheers,
> Matthieu
>
>
> >
> > thanks again!
> > Lavanya
> >
> > On Fri, 30 Nov 2007, Matthieu Riou wrote:
> >
> > > Hi,
> > >
> > > Sorry for taking so long but I've been side-tracked a couple of times
> > and
> > > this wasn't trivial to debug either. So I've found a couple of problems:
> > >
> > > 1. A potential deadlock (that was happening once in a while) in our JPA
> > > Process DAO. Basically each new instance creation resulted in an update
> > on
> > > the very same process record to increase some counter, which was pretty
> > bad.
> > >
> > > 2. A much more tricky bug with OpenJPA committing the transaction once
> > in a
> > > while without us asking for it. This appeared to be somewhat linked to
> > the
> > > Geronimo transaction manager as I've only seen that happening with it.
> > So
> > > once a while, when OpenJPA was getting a new set of sequence numbers to
> > > assign to primary keys, it forced a commit in the middle of our
> > transaction.
> > > Hence errors about not having a transaction anymore afterward. I've
> > upgraded
> > > to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly for
> > me
> > > know.
> > >
> > > I've done some tests on Tomcat/MySQL with 30 threads and it worked
> > properly.
> > > I think there's a lot we could do to improve performances by optimizing
> > the
> > > queries but I don't have much time for that right now. But at still it
> > works
> > > and it's not ridiculously slow with enough threads. So update your
> > version
> > > of the 1.1 branch and you should see an improvement.
> > >
> > > Cheers,
> > > Matthieu
> > >
> > > On Nov 28, 2007 5:28 PM, Matthieu Riou <ma...@offthelip.org> wrote:
> > >
> > > > Okay I can reproduce the problem, it's on Derby but I think in that
> > case
> > > > it doesn't make much difference. Seems to be a big database
> > deadlock... I'll
> > > > keep you posted.
> > > >
> > > > Cheers,
> > > > Matthieu
> > > >
> > > >
> > > > On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <laramakr@cs.indiana.edu
> > >
> > > > wrote:
> > > >
> > > > > Just to clarify, I am only trying to run 5 parallel clients but they
> > do
> > > > > make repeated helloWorld calls.
> > > > >
> > > > > I added these properties. But I am still getting the NPE. but in
> > > > > addition I am seeing another exception which might be a clue?
> > > > >
> > > > > The entire log is at
> > http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > > >
> > > > >
> > > > > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > > > > ERROR - GeronimoLog.error(108) | Method "run" in class
> > > > > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
> > > > > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > > > > synchronizer.
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer
> > (
> > > > > SimpleScheduler.java:206)
> > > > > at
> > > > >
> > org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > > > > MyRoleMessageExchangeImpl.java :215)
> > > > > at
> > > > > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > > > > MessageExchangeImpl.java:173)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > > > > BpelRuntimeContextImpl.java:544)
> > > > > at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > > > > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown
> > Source)
> > > > > at
> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > > DelegatingMethodAccessorImpl.java :25)
> > > > > at java.lang.reflect.Method.invoke(Method.java:585)
> > > > > at
> > > > > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java
> > :451)
> > > > > at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > > > > BpelRuntimeContextImpl.java:838)
> > > > > at
> > > > > org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > > > > PartnerLinkMyRoleImpl.java :197)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(
> > BpelProcess.java
> > > > > :167)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(
> > BpelProcess.java
> > > > > :335)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > > > > BpelEngineImpl.java:328)
> > > > > at
> > > > > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > > > > BpelServerImpl.java:365)
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > SimpleScheduler.java:341)
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > SimpleScheduler.java:340)
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > > > > SimpleScheduler.java:179)
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > SimpleScheduler.java:339)
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > SimpleScheduler.java:336)
> > > > > at
> > > > > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:269)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > > > > ThreadPoolExecutor.java:650)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > > > > ThreadPoolExecutor.java:675)
> > > > > at java.lang.Thread.run(Thread.java:595)
> > > > > Caused by: java.lang.NullPointerException
> > > > > at
> > > > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > > > > SimpleScheduler.java:194)
> > > > > ... 25 more
> > > > >
> > > > >
> > > > >
> > > > > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> > > > >
> > > > > > In internal mode you should adjust the pool size with something
> > like:
> > > > > >
> > > > > > ode-axis2.db.pool.max=100
> > > > > > ode-axis2.db.pool.min=40
> > > > > >
> > > > > > With 100 threads and the default pool size (10) you're *very*
> > probably
> > > > > > exhausting the number of connections.
> > > > > >
> > > > > > Cheers,
> > > > > > Matthieu
> > > > > >
> > > > > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > > > > laramakr@cs.indiana.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > > Lavanya, are you sure you configured a large enough connection
> > > > > pool to
> > > > > > > > handle all your requests? If you start a large number of
> > parrallel
> > > > > > > requests
> > > > > > > > they'll all try to get a DB connection and if there's not
> > enough
> > > > > of them
> > > > > > > > available, things run bad. Let me know.
> > > > > > >
> > > > > > > I also suspected a pool exhaustion. So based on earlier posts I
> > am
> > > > > using
> > > > > > > db.mode to be INTERNAL. How do I increase the db connection pool
> > in
> > > > > this
> > > > > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > > > > >
> > > > > > > Here is what I have so far in my ode-axis2.properites.
> > > > > > >
> > > > > > > ode-axis2.db.mode=INTERNAL
> > > > > > > ode-axis2.db.int.jdbcurl=***
> > > > > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > > > > ode-axis2.threads.pool.size=200
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
On Nov 30, 2007 5:10 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
wrote:
> Thanks! This is looking really good now!
>
> When I have a large number of connections (>75) I am beginning to see some
> TimeoutExceptions from the reply response. Is there a parameter I can
> tweak for the timeout value?
>
Do you know exactly what times out or have a stack trace? Could just be your
client connection.
Cheers,
Matthieu
>
> thanks again!
> Lavanya
>
> On Fri, 30 Nov 2007, Matthieu Riou wrote:
>
> > Hi,
> >
> > Sorry for taking so long but I've been side-tracked a couple of times
> and
> > this wasn't trivial to debug either. So I've found a couple of problems:
> >
> > 1. A potential deadlock (that was happening once in a while) in our JPA
> > Process DAO. Basically each new instance creation resulted in an update
> on
> > the very same process record to increase some counter, which was pretty
> bad.
> >
> > 2. A much more tricky bug with OpenJPA committing the transaction once
> in a
> > while without us asking for it. This appeared to be somewhat linked to
> the
> > Geronimo transaction manager as I've only seen that happening with it.
> So
> > once a while, when OpenJPA was getting a new set of sequence numbers to
> > assign to primary keys, it forced a commit in the middle of our
> transaction.
> > Hence errors about not having a transaction anymore afterward. I've
> upgraded
> > to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly for
> me
> > know.
> >
> > I've done some tests on Tomcat/MySQL with 30 threads and it worked
> properly.
> > I think there's a lot we could do to improve performances by optimizing
> the
> > queries but I don't have much time for that right now. But at still it
> works
> > and it's not ridiculously slow with enough threads. So update your
> version
> > of the 1.1 branch and you should see an improvement.
> >
> > Cheers,
> > Matthieu
> >
> > On Nov 28, 2007 5:28 PM, Matthieu Riou <ma...@offthelip.org> wrote:
> >
> > > Okay I can reproduce the problem, it's on Derby but I think in that
> case
> > > it doesn't make much difference. Seems to be a big database
> deadlock... I'll
> > > keep you posted.
> > >
> > > Cheers,
> > > Matthieu
> > >
> > >
> > > On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <laramakr@cs.indiana.edu
> >
> > > wrote:
> > >
> > > > Just to clarify, I am only trying to run 5 parallel clients but they
> do
> > > > make repeated helloWorld calls.
> > > >
> > > > I added these properties. But I am still getting the NPE. but in
> > > > addition I am seeing another exception which might be a clue?
> > > >
> > > > The entire log is at
> http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > >
> > > >
> > > > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > > > ERROR - GeronimoLog.error(108) | Method "run" in class
> > > > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
> > > > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > > > synchronizer.
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer
> (
> > > > SimpleScheduler.java:206)
> > > > at
> > > >
> org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > > > MyRoleMessageExchangeImpl.java :215)
> > > > at
> > > > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > > > MessageExchangeImpl.java:173)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > > > BpelRuntimeContextImpl.java:544)
> > > > at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > > > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown
> Source)
> > > > at
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > DelegatingMethodAccessorImpl.java :25)
> > > > at java.lang.reflect.Method.invoke(Method.java:585)
> > > > at
> > > > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java
> :451)
> > > > at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > > > BpelRuntimeContextImpl.java:838)
> > > > at
> > > > org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > > > PartnerLinkMyRoleImpl.java :197)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(
> BpelProcess.java
> > > > :167)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(
> BpelProcess.java
> > > > :335)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > > > BpelEngineImpl.java:328)
> > > > at
> > > > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > > > BpelServerImpl.java:365)
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > SimpleScheduler.java:341)
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > SimpleScheduler.java:340)
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > > > SimpleScheduler.java:179)
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > SimpleScheduler.java:339)
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > SimpleScheduler.java:336)
> > > > at
> > > > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:269)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > > > ThreadPoolExecutor.java:650)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > > > ThreadPoolExecutor.java:675)
> > > > at java.lang.Thread.run(Thread.java:595)
> > > > Caused by: java.lang.NullPointerException
> > > > at
> > > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > > > SimpleScheduler.java:194)
> > > > ... 25 more
> > > >
> > > >
> > > >
> > > > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> > > >
> > > > > In internal mode you should adjust the pool size with something
> like:
> > > > >
> > > > > ode-axis2.db.pool.max=100
> > > > > ode-axis2.db.pool.min=40
> > > > >
> > > > > With 100 threads and the default pool size (10) you're *very*
> probably
> > > > > exhausting the number of connections.
> > > > >
> > > > > Cheers,
> > > > > Matthieu
> > > > >
> > > > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > > > laramakr@cs.indiana.edu>
> > > > > wrote:
> > > > >
> > > > > > > Lavanya, are you sure you configured a large enough connection
> > > > pool to
> > > > > > > handle all your requests? If you start a large number of
> parrallel
> > > > > > requests
> > > > > > > they'll all try to get a DB connection and if there's not
> enough
> > > > of them
> > > > > > > available, things run bad. Let me know.
> > > > > >
> > > > > > I also suspected a pool exhaustion. So based on earlier posts I
> am
> > > > using
> > > > > > db.mode to be INTERNAL. How do I increase the db connection pool
> in
> > > > this
> > > > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > > > >
> > > > > > Here is what I have so far in my ode-axis2.properites.
> > > > > >
> > > > > > ode-axis2.db.mode=INTERNAL
> > > > > > ode-axis2.db.int.jdbcurl=***
> > > > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > > > ode-axis2.threads.pool.size=200
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
Thanks! This is looking really good now!
When I have a large number of connections (>75) I am beginning to see some
TimeoutExceptions from the reply response. Is there a parameter I can
tweak for the timeout value?
thanks again!
Lavanya
On Fri, 30 Nov 2007, Matthieu Riou wrote:
> Hi,
>
> Sorry for taking so long but I've been side-tracked a couple of times and
> this wasn't trivial to debug either. So I've found a couple of problems:
>
> 1. A potential deadlock (that was happening once in a while) in our JPA
> Process DAO. Basically each new instance creation resulted in an update on
> the very same process record to increase some counter, which was pretty bad.
>
> 2. A much more tricky bug with OpenJPA committing the transaction once in a
> while without us asking for it. This appeared to be somewhat linked to the
> Geronimo transaction manager as I've only seen that happening with it. So
> once a while, when OpenJPA was getting a new set of sequence numbers to
> assign to primary keys, it forced a commit in the middle of our transaction.
> Hence errors about not having a transaction anymore afterward. I've upgraded
> to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly for me
> know.
>
> I've done some tests on Tomcat/MySQL with 30 threads and it worked properly.
> I think there's a lot we could do to improve performances by optimizing the
> queries but I don't have much time for that right now. But at still it works
> and it's not ridiculously slow with enough threads. So update your version
> of the 1.1 branch and you should see an improvement.
>
> Cheers,
> Matthieu
>
> On Nov 28, 2007 5:28 PM, Matthieu Riou <ma...@offthelip.org> wrote:
>
> > Okay I can reproduce the problem, it's on Derby but I think in that case
> > it doesn't make much difference. Seems to be a big database deadlock... I'll
> > keep you posted.
> >
> > Cheers,
> > Matthieu
> >
> >
> > On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> > wrote:
> >
> > > Just to clarify, I am only trying to run 5 parallel clients but they do
> > > make repeated helloWorld calls.
> > >
> > > I added these properties. But I am still getting the NPE. but in
> > > addition I am seeing another exception which might be a clue?
> > >
> > > The entire log is at http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > >
> > >
> > > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > > ERROR - GeronimoLog.error(108) | Method "run" in class
> > > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
> > > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > > synchronizer.
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > > SimpleScheduler.java:206)
> > > at
> > > org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > > MyRoleMessageExchangeImpl.java :215)
> > > at
> > > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > > MessageExchangeImpl.java:173)
> > > at
> > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > > BpelRuntimeContextImpl.java:544)
> > > at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java :25)
> > > at java.lang.reflect.Method.invoke(Method.java:585)
> > > at
> > > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451)
> > > at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java :139)
> > > at
> > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > > BpelRuntimeContextImpl.java:838)
> > > at
> > > org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > > PartnerLinkMyRoleImpl.java :197)
> > > at
> > > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java
> > > :167)
> > > at
> > > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(BpelProcess.java
> > > :335)
> > > at
> > > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > > BpelEngineImpl.java:328)
> > > at
> > > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > > BpelServerImpl.java:365)
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > SimpleScheduler.java:341)
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > SimpleScheduler.java:340)
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > > SimpleScheduler.java:179)
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > SimpleScheduler.java:339)
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > SimpleScheduler.java:336)
> > > at
> > > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:269)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > > ThreadPoolExecutor.java:650)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > > ThreadPoolExecutor.java:675)
> > > at java.lang.Thread.run(Thread.java:595)
> > > Caused by: java.lang.NullPointerException
> > > at
> > > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer (
> > > SimpleScheduler.java:194)
> > > ... 25 more
> > >
> > >
> > >
> > > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> > >
> > > > In internal mode you should adjust the pool size with something like:
> > > >
> > > > ode-axis2.db.pool.max=100
> > > > ode-axis2.db.pool.min=40
> > > >
> > > > With 100 threads and the default pool size (10) you're *very* probably
> > > > exhausting the number of connections.
> > > >
> > > > Cheers,
> > > > Matthieu
> > > >
> > > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > > laramakr@cs.indiana.edu>
> > > > wrote:
> > > >
> > > > > > Lavanya, are you sure you configured a large enough connection
> > > pool to
> > > > > > handle all your requests? If you start a large number of parrallel
> > > > > requests
> > > > > > they'll all try to get a DB connection and if there's not enough
> > > of them
> > > > > > available, things run bad. Let me know.
> > > > >
> > > > > I also suspected a pool exhaustion. So based on earlier posts I am
> > > using
> > > > > db.mode to be INTERNAL. How do I increase the db connection pool in
> > > this
> > > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > > >
> > > > > Here is what I have so far in my ode-axis2.properites.
> > > > >
> > > > > ode-axis2.db.mode=INTERNAL
> > > > > ode-axis2.db.int.jdbcurl=***
> > > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > > ode-axis2.threads.pool.size=200
> > > > >
> > > > >
> > > >
> > >
> >
> >
>
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
Hi,
Sorry for taking so long but I've been side-tracked a couple of times and
this wasn't trivial to debug either. So I've found a couple of problems:
1. A potential deadlock (that was happening once in a while) in our JPA
Process DAO. Basically each new instance creation resulted in an update on
the very same process record to increase some counter, which was pretty bad.
2. A much more tricky bug with OpenJPA committing the transaction once in a
while without us asking for it. This appeared to be somewhat linked to the
Geronimo transaction manager as I've only seen that happening with it. So
once a while, when OpenJPA was getting a new set of sequence numbers to
assign to primary keys, it forced a commit in the middle of our transaction.
Hence errors about not having a transaction anymore afterward. I've upgraded
to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly for me
know.
I've done some tests on Tomcat/MySQL with 30 threads and it worked properly.
I think there's a lot we could do to improve performances by optimizing the
queries but I don't have much time for that right now. But at still it works
and it's not ridiculously slow with enough threads. So update your version
of the 1.1 branch and you should see an improvement.
Cheers,
Matthieu
On Nov 28, 2007 5:28 PM, Matthieu Riou <ma...@offthelip.org> wrote:
> Okay I can reproduce the problem, it's on Derby but I think in that case
> it doesn't make much difference. Seems to be a big database deadlock... I'll
> keep you posted.
>
> Cheers,
> Matthieu
>
>
> On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> wrote:
>
> > Just to clarify, I am only trying to run 5 parallel clients but they do
> > make repeated helloWorld calls.
> >
> > I added these properties. But I am still getting the NPE. but in
> > addition I am seeing another exception which might be a clue?
> >
> > The entire log is at http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> >
> >
> > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > ERROR - GeronimoLog.error(108) | Method "run" in class
> > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
> > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > synchronizer.
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > SimpleScheduler.java:206)
> > at
> > org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > MyRoleMessageExchangeImpl.java :215)
> > at
> > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > MessageExchangeImpl.java:173)
> > at
> > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > BpelRuntimeContextImpl.java:544)
> > at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java :25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451)
> > at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java :139)
> > at
> > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > BpelRuntimeContextImpl.java:838)
> > at
> > org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > PartnerLinkMyRoleImpl.java :197)
> > at
> > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java
> > :167)
> > at
> > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(BpelProcess.java
> > :335)
> > at
> > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > BpelEngineImpl.java:328)
> > at
> > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > BpelServerImpl.java:365)
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > SimpleScheduler.java:341)
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > SimpleScheduler.java:340)
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > SimpleScheduler.java:179)
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > SimpleScheduler.java:339)
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > SimpleScheduler.java:336)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java:269)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > ThreadPoolExecutor.java:650)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > ThreadPoolExecutor.java:675)
> > at java.lang.Thread.run(Thread.java:595)
> > Caused by: java.lang.NullPointerException
> > at
> > org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer (
> > SimpleScheduler.java:194)
> > ... 25 more
> >
> >
> >
> > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> >
> > > In internal mode you should adjust the pool size with something like:
> > >
> > > ode-axis2.db.pool.max=100
> > > ode-axis2.db.pool.min=40
> > >
> > > With 100 threads and the default pool size (10) you're *very* probably
> > > exhausting the number of connections.
> > >
> > > Cheers,
> > > Matthieu
> > >
> > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > laramakr@cs.indiana.edu>
> > > wrote:
> > >
> > > > > Lavanya, are you sure you configured a large enough connection
> > pool to
> > > > > handle all your requests? If you start a large number of parrallel
> > > > requests
> > > > > they'll all try to get a DB connection and if there's not enough
> > of them
> > > > > available, things run bad. Let me know.
> > > >
> > > > I also suspected a pool exhaustion. So based on earlier posts I am
> > using
> > > > db.mode to be INTERNAL. How do I increase the db connection pool in
> > this
> > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > >
> > > > Here is what I have so far in my ode-axis2.properites.
> > > >
> > > > ode-axis2.db.mode=INTERNAL
> > > > ode-axis2.db.int.jdbcurl=***
> > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > ode-axis2.threads.pool.size=200
> > > >
> > > >
> > >
> >
>
>
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
Okay I can reproduce the problem, it's on Derby but I think in that case it
doesn't make much difference. Seems to be a big database deadlock... I'll
keep you posted.
Cheers,
Matthieu
On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
wrote:
> Just to clarify, I am only trying to run 5 parallel clients but they do
> make repeated helloWorld calls.
>
> I added these properties. But I am still getting the NPE. but in
> addition I am seeing another exception which might be a clue?
>
> The entire log is at http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
>
>
> 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> ERROR - GeronimoLog.error(108) | Method "run" in class
> "org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
> org.apache.ode.bpel.iapi.ContextException: Unable to register
> synchronizer.
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> SimpleScheduler.java:206)
> at
> org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> MyRoleMessageExchangeImpl.java:215)
> at
> org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> MessageExchangeImpl.java:173)
> at
> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> BpelRuntimeContextImpl.java:544)
> at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451)
> at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
> at
> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> BpelRuntimeContextImpl.java:838)
> at
> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> PartnerLinkMyRoleImpl.java:197)
> at
> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:167)
> at
> org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(BpelProcess.java
> :335)
> at
> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(
> BpelEngineImpl.java:328)
> at
> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> BpelServerImpl.java:365)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> SimpleScheduler.java:341)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> SimpleScheduler.java:340)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> SimpleScheduler.java:179)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> SimpleScheduler.java:339)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> SimpleScheduler.java:336)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
> :675)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.lang.NullPointerException
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> SimpleScheduler.java:194)
> ... 25 more
>
>
>
> On Wed, 28 Nov 2007, Matthieu Riou wrote:
>
> > In internal mode you should adjust the pool size with something like:
> >
> > ode-axis2.db.pool.max=100
> > ode-axis2.db.pool.min=40
> >
> > With 100 threads and the default pool size (10) you're *very* probably
> > exhausting the number of connections.
> >
> > Cheers,
> > Matthieu
> >
> > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> > wrote:
> >
> > > > Lavanya, are you sure you configured a large enough connection pool
> to
> > > > handle all your requests? If you start a large number of parrallel
> > > requests
> > > > they'll all try to get a DB connection and if there's not enough of
> them
> > > > available, things run bad. Let me know.
> > >
> > > I also suspected a pool exhaustion. So based on earlier posts I am
> using
> > > db.mode to be INTERNAL. How do I increase the db connection pool in
> this
> > > mode? I looked in the FAQ etc but didn't find a mention.
> > >
> > > Here is what I have so far in my ode-axis2.properites.
> > >
> > > ode-axis2.db.mode=INTERNAL
> > > ode-axis2.db.int.jdbcurl=***
> > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > ode-axis2.threads.pool.size=200
> > >
> > >
> >
>
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
Just to clarify, I am only trying to run 5 parallel clients but they do
make repeated helloWorld calls.
I added these properties. But I am still getting the NPE. but in
addition I am seeing another exception which might be a clue?
The entire log is at http://www.cs.indiana.edu/~laramakr/catalina.out.1
15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
ERROR - GeronimoLog.error(108) | Method "run" in class
"org.apache.ode.bpel.runtime.REPLY" threw an unexpected exception.
org.apache.ode.bpel.iapi.ContextException: Unable to register
synchronizer.
at
org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(SimpleScheduler.java:206)
at
org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(MyRoleMessageExchangeImpl.java:215)
at
org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(MessageExchangeImpl.java:173)
at
org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(BpelRuntimeContextImpl.java:544)
at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451)
at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139)
at
org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntimeContextImpl.java:838)
at
org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(PartnerLinkMyRoleImpl.java:197)
at
org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:167)
at
org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(BpelProcess.java:335)
at
org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:328)
at
org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.java:365)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler.java:341)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler.java:340)
at
org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:179)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.java:339)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.java:336)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.NullPointerException
at
org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(SimpleScheduler.java:194)
... 25 more
On Wed, 28 Nov 2007, Matthieu Riou wrote:
> In internal mode you should adjust the pool size with something like:
>
> ode-axis2.db.pool.max=100
> ode-axis2.db.pool.min=40
>
> With 100 threads and the default pool size (10) you're *very* probably
> exhausting the number of connections.
>
> Cheers,
> Matthieu
>
> On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> wrote:
>
> > > Lavanya, are you sure you configured a large enough connection pool to
> > > handle all your requests? If you start a large number of parrallel
> > requests
> > > they'll all try to get a DB connection and if there's not enough of them
> > > available, things run bad. Let me know.
> >
> > I also suspected a pool exhaustion. So based on earlier posts I am using
> > db.mode to be INTERNAL. How do I increase the db connection pool in this
> > mode? I looked in the FAQ etc but didn't find a mention.
> >
> > Here is what I have so far in my ode-axis2.properites.
> >
> > ode-axis2.db.mode=INTERNAL
> > ode-axis2.db.int.jdbcurl=***
> > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > ode-axis2.threads.pool.size=200
> >
> >
>
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
In internal mode you should adjust the pool size with something like:
ode-axis2.db.pool.max=100
ode-axis2.db.pool.min=40
With 100 threads and the default pool size (10) you're *very* probably
exhausting the number of connections.
Cheers,
Matthieu
On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
wrote:
> > Lavanya, are you sure you configured a large enough connection pool to
> > handle all your requests? If you start a large number of parrallel
> requests
> > they'll all try to get a DB connection and if there's not enough of them
> > available, things run bad. Let me know.
>
> I also suspected a pool exhaustion. So based on earlier posts I am using
> db.mode to be INTERNAL. How do I increase the db connection pool in this
> mode? I looked in the FAQ etc but didn't find a mention.
>
> Here is what I have so far in my ode-axis2.properites.
>
> ode-axis2.db.mode=INTERNAL
> ode-axis2.db.int.jdbcurl=***
> ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> ode-axis2.threads.pool.size=200
>
>
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
> Lavanya, are you sure you configured a large enough connection pool to
> handle all your requests? If you start a large number of parrallel requests
> they'll all try to get a DB connection and if there's not enough of them
> available, things run bad. Let me know.
I also suspected a pool exhaustion. So based on earlier posts I am using
db.mode to be INTERNAL. How do I increase the db connection pool in this
mode? I looked in the FAQ etc but didn't find a mention.
Here is what I have so far in my ode-axis2.properites.
ode-axis2.db.mode=INTERNAL
ode-axis2.db.int.jdbcurl=***
ode-axis2.db.int.driver=com.mysql.jdbc.Driver
ode-axis2.threads.pool.size=200
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
On Nov 28, 2007 10:03 AM, Paul Brown <pa...@gmail.com> wrote:
>
> This bit:
>
> Scheduled job failed; jobDetail={type=INVOKE_INTERNAL,
> mexid=hqejbhcnphr2s7en3xcq1k, pid={http://ode/bpel/unit-test}
> HelloWorld2-3}
> java.lang.NullPointerException
> at
> org.apache.ode.dao.jpa.BPELDAOConnectionFactoryImpl.getConnection
> (BPELDAOConnectionFactoryImpl.java:63)
>
> Looks interesting. Either no transaction manager or no transaction...?
>
I'm actually leaning toward pool exhaustion, although it's hard to be sure.
Lavanya, are you sure you configured a large enough connection pool to
handle all your requests? If you start a large number of parrallel requests
they'll all try to get a DB connection and if there's not enough of them
available, things run bad. Let me know.
Cheers,
Matthieu
>
> -- Paul
>
> On Nov 28, 2007, at 9:57 AM, Lavanya Ramakrishnan wrote:
>
> > Yes, I am running with the latest version of the branch as of
> > yesterday. I
> > didnt see any errors in the log before the NullPointerException.
> >
> > I have put up the log (with DEBUG on):
> >
> > http://www.cs.indiana.edu/~laramakr/catalina.out<http://www.cs.indiana.edu/%7Elaramakr/catalina.out>
> >
> >
> >
> > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> >
> >> Hi Lavanaya,
> >>
> >> Just to confirm, you are running with the latest version of the
> >> branch,
> >> right?
> >>
> >> There's probably a fault or an exception happening inside the
> >> engine that
> >> causes it to provide an empty reply to the incoming message. Could
> >> you check
> >> whether you have a previous fault or exception message in your
> >> logs? Also if
> >> you could provide a full log with debug turned on that would help.
> >>
> >> Thanks!
> >> Matthieu
> >>
> >> On Nov 27, 2007 2:21 PM, Lavanya Ramakrishnan
> >> <la...@cs.indiana.edu>
> >> wrote:
> >>
> >>> I am running a simple test with multiple concurrent clients that
> >>> make
> >>> repeated calls to the engine and getting the NullPointerException
> >>> below.
> >>>
> >>> My configuration is
> >>>
> >>> Tomcat 5.5.23
> >>> MySQL for backend
> >>> ODE 1.1 subversion branch (only change is the patch Jira ODE-211
> >>> applied)
> >>>
> >>> Example: HelloWorld
> >>>
> >>> I noticed some previous messages with the same error which seemed to
> >>> indicate a problem with the axis-kernel jar. I noticed that the
> >>> 1.1 branch
> >>> already has the 1.3 jar which is supposed to have fixed the problem.
> >>>
> >>> The problem seems to definitely get worse with increasing concurrent
> >>> clients. With 5 clients and 50 back to back calls a large number
> >>> of them
> >>> fail.
> >>>
> >>> Any suggestions will be helpful.
> >>>
> >>> thanks!
> >>> Lavanya
> >>>
> >>> ERROR - GeronimoLog.error(108) | Error processing response for MEX
> >>> {MyRoleMex#hqejbhcnphr2s4q4a7l7qd [Client hqejbhcnphr2s4q4a7l7qc]
> >>> calling
> >>> {http://ode/bpel/unit-test.wsdl}HelloService.hello(...)<http://ode/bpel/unit-test.wsdl%7DHelloService.hello%28...%29>
> <http://
> >>> ode/bpel/unit-test.wsdl%7DHelloService.hello%28...%29>
> >>> }
> >>> java.lang.NullPointerException: null message!
> >>> at
> >>> org.apache.ode.bpel.engine.MessageImpl.<init>(MessageImpl.java:40)
> >>> at
> >>> org.apache.ode.bpel.engine.MessageExchangeImpl.getResponse(
> >>> MessageExchangeImpl.java:97)
> >>> at org.apache.ode.axis2.ODEService.onResponse
> >>> (ODEService.java:226)
> >>> at
> >>> org.apache.ode.axis2.ODEService.onAxisMessageExchange
> >>> (ODEService.java:172)
> >>> at
> >>> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> >>> ODEMessageReceiver.java:69)
> >>> at
> >>> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> >>> ODEMessageReceiver.java:50)
> >>> at
> >>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(
> >>> AbstractMessageReceiver.java:96)
> >>> at org.apache.axis2.engine.AxisEngine.receive
> >>> (AxisEngine.java:145)
> >>> at
> >>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRe
> >>> quest(
> >>> HTTPTransportUtils.java:275)
> >>> at
> >>> org.apache.axis2.transport.http.AxisServlet.doPost
> >>> (AxisServlet.java:120)
> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
> >>> 710)
> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
> >>> 803)
> >>> at
> >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> >>> ApplicationFilterChain.java:269)
> >>> at
> >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(
> >>> ApplicationFilterChain.java:188)
> >>> at
> >>> org.apache.catalina.core.StandardWrapperValve.invoke(
> >>> StandardWrapperValve.java:210)
> >>> at
> >>> org.apache.catalina.core.StandardContextValve.invoke(
> >>> StandardContextValve.java:174)
> >>> at
> >>> org.apache.catalina.core.StandardHostValve.invoke
> >>> (StandardHostValve.java
> >>> :127)
> >>> at
> >>> org.apache.catalina.valves.ErrorReportValve.invoke
> >>> (ErrorReportValve.java
> >>> :117)
> >>> at
> >>> org.apache.catalina.core.StandardEngineValve.invoke(
> >>> StandardEngineValve.java:108)
> >>> at
> >>> org.apache.catalina.connector.CoyoteAdapter.service
> >>> (CoyoteAdapter.java
> >>> :151)
> >>> at
> >>> org.apache.coyote.http11.Http11Processor.process
> >>> (Http11Processor.java:870)
> >>> at
> >>>
> >>> org.apache.coyote.http11.Http11BaseProtocol
> >>> $Http11ConnectionHandler.processConnection
> >>> (Http11BaseProtocol.java:665)
> >>> at
> >>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
> >>> PoolTcpEndpoint.java:528)
> >>> at
> >>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
> >>> LeaderFollowerWorkerThread.java:81)
> >>> at
> >>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> >>> ThreadPool.java:685)
> >>> at java.lang.Thread.run(Thread.java:595)
> >>>
> >>>
> >>
>
> Paul Brown
> paulrbrown@gmail.com
>
>
>
>
Re: Problem with multiple concurrent clients
Posted by Paul Brown <pa...@gmail.com>.
This bit:
Scheduled job failed; jobDetail={type=INVOKE_INTERNAL,
mexid=hqejbhcnphr2s7en3xcq1k, pid={http://ode/bpel/unit-test}
HelloWorld2-3}
java.lang.NullPointerException
at org.apache.ode.dao.jpa.BPELDAOConnectionFactoryImpl.getConnection
(BPELDAOConnectionFactoryImpl.java:63)
Looks interesting. Either no transaction manager or no transaction...?
-- Paul
On Nov 28, 2007, at 9:57 AM, Lavanya Ramakrishnan wrote:
> Yes, I am running with the latest version of the branch as of
> yesterday. I
> didnt see any errors in the log before the NullPointerException.
>
> I have put up the log (with DEBUG on):
>
> http://www.cs.indiana.edu/~laramakr/catalina.out
>
>
>
> On Wed, 28 Nov 2007, Matthieu Riou wrote:
>
>> Hi Lavanaya,
>>
>> Just to confirm, you are running with the latest version of the
>> branch,
>> right?
>>
>> There's probably a fault or an exception happening inside the
>> engine that
>> causes it to provide an empty reply to the incoming message. Could
>> you check
>> whether you have a previous fault or exception message in your
>> logs? Also if
>> you could provide a full log with debug turned on that would help.
>>
>> Thanks!
>> Matthieu
>>
>> On Nov 27, 2007 2:21 PM, Lavanya Ramakrishnan
>> <la...@cs.indiana.edu>
>> wrote:
>>
>>> I am running a simple test with multiple concurrent clients that
>>> make
>>> repeated calls to the engine and getting the NullPointerException
>>> below.
>>>
>>> My configuration is
>>>
>>> Tomcat 5.5.23
>>> MySQL for backend
>>> ODE 1.1 subversion branch (only change is the patch Jira ODE-211
>>> applied)
>>>
>>> Example: HelloWorld
>>>
>>> I noticed some previous messages with the same error which seemed to
>>> indicate a problem with the axis-kernel jar. I noticed that the
>>> 1.1 branch
>>> already has the 1.3 jar which is supposed to have fixed the problem.
>>>
>>> The problem seems to definitely get worse with increasing concurrent
>>> clients. With 5 clients and 50 back to back calls a large number
>>> of them
>>> fail.
>>>
>>> Any suggestions will be helpful.
>>>
>>> thanks!
>>> Lavanya
>>>
>>> ERROR - GeronimoLog.error(108) | Error processing response for MEX
>>> {MyRoleMex#hqejbhcnphr2s4q4a7l7qd [Client hqejbhcnphr2s4q4a7l7qc]
>>> calling
>>> {http://ode/bpel/unit-test.wsdl}HelloService.hello(...)<http://
>>> ode/bpel/unit-test.wsdl%7DHelloService.hello%28...%29>
>>> }
>>> java.lang.NullPointerException: null message!
>>> at
>>> org.apache.ode.bpel.engine.MessageImpl.<init>(MessageImpl.java:40)
>>> at
>>> org.apache.ode.bpel.engine.MessageExchangeImpl.getResponse(
>>> MessageExchangeImpl.java:97)
>>> at org.apache.ode.axis2.ODEService.onResponse
>>> (ODEService.java:226)
>>> at
>>> org.apache.ode.axis2.ODEService.onAxisMessageExchange
>>> (ODEService.java:172)
>>> at
>>> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
>>> ODEMessageReceiver.java:69)
>>> at
>>> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
>>> ODEMessageReceiver.java:50)
>>> at
>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(
>>> AbstractMessageReceiver.java:96)
>>> at org.apache.axis2.engine.AxisEngine.receive
>>> (AxisEngine.java:145)
>>> at
>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRe
>>> quest(
>>> HTTPTransportUtils.java:275)
>>> at
>>> org.apache.axis2.transport.http.AxisServlet.doPost
>>> (AxisServlet.java:120)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
>>> 710)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:
>>> 803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>>> ApplicationFilterChain.java:269)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(
>>> ApplicationFilterChain.java:188)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(
>>> StandardWrapperValve.java:210)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(
>>> StandardContextValve.java:174)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke
>>> (StandardHostValve.java
>>> :127)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke
>>> (ErrorReportValve.java
>>> :117)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(
>>> StandardEngineValve.java:108)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service
>>> (CoyoteAdapter.java
>>> :151)
>>> at
>>> org.apache.coyote.http11.Http11Processor.process
>>> (Http11Processor.java:870)
>>> at
>>>
>>> org.apache.coyote.http11.Http11BaseProtocol
>>> $Http11ConnectionHandler.processConnection
>>> (Http11BaseProtocol.java:665)
>>> at
>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
>>> PoolTcpEndpoint.java:528)
>>> at
>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
>>> LeaderFollowerWorkerThread.java:81)
>>> at
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
>>> ThreadPool.java:685)
>>> at java.lang.Thread.run(Thread.java:595)
>>>
>>>
>>
Paul Brown
paulrbrown@gmail.com
Re: Problem with multiple concurrent clients
Posted by Lavanya Ramakrishnan <la...@cs.indiana.edu>.
Yes, I am running with the latest version of the branch as of yesterday. I
didnt see any errors in the log before the NullPointerException.
I have put up the log (with DEBUG on):
http://www.cs.indiana.edu/~laramakr/catalina.out
On Wed, 28 Nov 2007, Matthieu Riou wrote:
> Hi Lavanaya,
>
> Just to confirm, you are running with the latest version of the branch,
> right?
>
> There's probably a fault or an exception happening inside the engine that
> causes it to provide an empty reply to the incoming message. Could you check
> whether you have a previous fault or exception message in your logs? Also if
> you could provide a full log with debug turned on that would help.
>
> Thanks!
> Matthieu
>
> On Nov 27, 2007 2:21 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
> wrote:
>
> > I am running a simple test with multiple concurrent clients that make
> > repeated calls to the engine and getting the NullPointerException below.
> >
> > My configuration is
> >
> > Tomcat 5.5.23
> > MySQL for backend
> > ODE 1.1 subversion branch (only change is the patch Jira ODE-211 applied)
> >
> > Example: HelloWorld
> >
> > I noticed some previous messages with the same error which seemed to
> > indicate a problem with the axis-kernel jar. I noticed that the 1.1 branch
> > already has the 1.3 jar which is supposed to have fixed the problem.
> >
> > The problem seems to definitely get worse with increasing concurrent
> > clients. With 5 clients and 50 back to back calls a large number of them
> > fail.
> >
> > Any suggestions will be helpful.
> >
> > thanks!
> > Lavanya
> >
> > ERROR - GeronimoLog.error(108) | Error processing response for MEX
> > {MyRoleMex#hqejbhcnphr2s4q4a7l7qd [Client hqejbhcnphr2s4q4a7l7qc] calling
> > {http://ode/bpel/unit-test.wsdl}HelloService.hello(...)<http://ode/bpel/unit-test.wsdl%7DHelloService.hello%28...%29>
> > }
> > java.lang.NullPointerException: null message!
> > at
> > org.apache.ode.bpel.engine.MessageImpl.<init>(MessageImpl.java:40)
> > at
> > org.apache.ode.bpel.engine.MessageExchangeImpl.getResponse(
> > MessageExchangeImpl.java:97)
> > at org.apache.ode.axis2.ODEService.onResponse(ODEService.java:226)
> > at
> > org.apache.ode.axis2.ODEService.onAxisMessageExchange(ODEService.java:172)
> > at
> > org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> > ODEMessageReceiver.java:69)
> > at
> > org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> > ODEMessageReceiver.java:50)
> > at
> > org.apache.axis2.receivers.AbstractMessageReceiver.receive(
> > AbstractMessageReceiver.java:96)
> > at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)
> > at
> > org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(
> > HTTPTransportUtils.java:275)
> > at
> > org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> > ApplicationFilterChain.java:269)
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(
> > ApplicationFilterChain.java:188)
> > at
> > org.apache.catalina.core.StandardWrapperValve.invoke(
> > StandardWrapperValve.java:210)
> > at
> > org.apache.catalina.core.StandardContextValve.invoke(
> > StandardContextValve.java:174)
> > at
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
> > :127)
> > at
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
> > :117)
> > at
> > org.apache.catalina.core.StandardEngineValve.invoke(
> > StandardEngineValve.java:108)
> > at
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> > :151)
> > at
> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
> > at
> >
> > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
> > (Http11BaseProtocol.java:665)
> > at
> > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
> > PoolTcpEndpoint.java:528)
> > at
> > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
> > LeaderFollowerWorkerThread.java:81)
> > at
> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> > ThreadPool.java:685)
> > at java.lang.Thread.run(Thread.java:595)
> >
> >
>
Re: Problem with multiple concurrent clients
Posted by Matthieu Riou <ma...@offthelip.org>.
Hi Lavanaya,
Just to confirm, you are running with the latest version of the branch,
right?
There's probably a fault or an exception happening inside the engine that
causes it to provide an empty reply to the incoming message. Could you check
whether you have a previous fault or exception message in your logs? Also if
you could provide a full log with debug turned on that would help.
Thanks!
Matthieu
On Nov 27, 2007 2:21 PM, Lavanya Ramakrishnan <la...@cs.indiana.edu>
wrote:
> I am running a simple test with multiple concurrent clients that make
> repeated calls to the engine and getting the NullPointerException below.
>
> My configuration is
>
> Tomcat 5.5.23
> MySQL for backend
> ODE 1.1 subversion branch (only change is the patch Jira ODE-211 applied)
>
> Example: HelloWorld
>
> I noticed some previous messages with the same error which seemed to
> indicate a problem with the axis-kernel jar. I noticed that the 1.1 branch
> already has the 1.3 jar which is supposed to have fixed the problem.
>
> The problem seems to definitely get worse with increasing concurrent
> clients. With 5 clients and 50 back to back calls a large number of them
> fail.
>
> Any suggestions will be helpful.
>
> thanks!
> Lavanya
>
> ERROR - GeronimoLog.error(108) | Error processing response for MEX
> {MyRoleMex#hqejbhcnphr2s4q4a7l7qd [Client hqejbhcnphr2s4q4a7l7qc] calling
> {http://ode/bpel/unit-test.wsdl}HelloService.hello(...)<http://ode/bpel/unit-test.wsdl%7DHelloService.hello%28...%29>
> }
> java.lang.NullPointerException: null message!
> at
> org.apache.ode.bpel.engine.MessageImpl.<init>(MessageImpl.java:40)
> at
> org.apache.ode.bpel.engine.MessageExchangeImpl.getResponse(
> MessageExchangeImpl.java:97)
> at org.apache.ode.axis2.ODEService.onResponse(ODEService.java:226)
> at
> org.apache.ode.axis2.ODEService.onAxisMessageExchange(ODEService.java:172)
> at
> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> ODEMessageReceiver.java:69)
> at
> org.apache.ode.axis2.hooks.ODEMessageReceiver.invokeBusinessLogic(
> ODEMessageReceiver.java:50)
> at
> org.apache.axis2.receivers.AbstractMessageReceiver.receive(
> AbstractMessageReceiver.java:96)
> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145)
> at
> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(
> HTTPTransportUtils.java:275)
> at
> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:269)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:210)
> at
> org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:174)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
> :127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
> :117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> :151)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
> at
>
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
> (Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
> PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
> LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
> ThreadPool.java:685)
> at java.lang.Thread.run(Thread.java:595)
>
>