You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ode.apache.org by "Nowakowski, Mateusz" <Ma...@sabre-holdings.com> on 2007/12/04 12:34:38 UTC
The execution queue is backed up... Forcing ContextException + MEMORY LEAK ??
Yes, I use 120 threads for ode and it is probably maximum considering
that every service mix components need to have similar own thread pool.
I launched longer stress test for 100 clients and for 50 clients. For
100 clients application crashed after 3h.
For 50 clients application survives 10h stress test, but gc log indicate
slow but memory leak. I investigate it, it can be service mix...
but I see the same exception all the time:
15:02:43,927 | ERROR | pool-10-thread-114 | SimpleScheduler |
duler.simple.SimpleScheduler$4 353 | Error while executing transaction
org.apache.ode.bpel.iapi.ContextException: The execution queue is backed
up... Forcing ContextException
at
org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
pleScheduler.java:227)
at
org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
java:290)
at
org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
java:373)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
.java:341)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
.java:340)
at
org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
heduler.java:179)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
ava:339)
at
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
ava:336)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:675)
at java.lang.Thread.run(Thread.java:595)
It is impossible to fill ode thread poll due to it is set to 120
threads, and the maximum parallel requests are 50.
I dig further and see that SimpleScheduler uses hardcoded size of
concurrent task which is 200. Still maximum concurrent requests is 50.
What is done with the request when this exception is thrown ??
How to get rid of this ??
Please help.
--
Regards
Mateusz Nowakowski
-----Original Message-----
From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On Behalf
Of Matthieu Riou
Sent: Thursday, November 29, 2007 5:21 PM
To: dev@ode.apache.org
Subject: Re: The execution queue is backed up... Forcing
ContextException
Do you saturate your CPU (about 85-90%)? If not, more threads should
bring
you more performance at least for in-memory. For persistent processes,
it's
the database optimization game...
Cheers,
Matthieu
On Nov 29, 2007 8:01 AM, Nowakowski, Mateusz <
Mateusz.Nowakowski@sabre-holdings.com> wrote:
> Thanks. It helps a bit. Application doesn't crash anymore but
> performance is decreased than for 50 parallel clients... But it can be
> different reason (not enough RAM or CPU)
>
> I know about this property before but I don't use because I assume
that
> BPEL caches threads and creates new thread when necessary.
> I looked on ODEService.initBpelServer() and saw:
>
> if (_ode._config.getThreadPoolMaxSize() == 0)
> _ode._executorService = Executors.newCachedThreadPool();
> else
> _ode._executorService =
> Executors.newFixedThreadPool(_ode._config.getThreadPoolMaxSize());
>
>
> --
> Regards
> Mateusz Nowakowski
>
> -----Original Message-----
> From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On
Behalf
> Of Matthieu Riou
> Sent: Wednesday, November 28, 2007 5:17 PM
> To: dev@ode.apache.org
> Subject: Re: The execution queue is backed up... Forcing
> ContextException
>
> Hi Mateusz,
>
> I've guessing you're exhausting ODE's thread pool. When the engine
can't
> keep up anymore and messages are coming faster that they can be
> consumed,
> the execution queue gets backed up and instead of accumulating
messages
> we
> throw an exception.
>
> To increase the size of the pool, you can set the following property
in
> ODE's configuration:
>
> ode-jbi.threads.pool.size=100
>
> Let us know how it goes.
>
> Cheers,
> Matthieu
>
> On Nov 28, 2007 6:56 AM, Nowakowski, Mateusz <
> Mateusz.Nowakowski@sabre-holdings.com> wrote:
>
> > I've performing some performance tests using Fuse 3.3.0.2 and Apache
> ODE
> > 1.1.
> >
> > I tested it using 100 parallel clients.
> > I used such thread pool properties:
> > servicemix.corePoolSize = 32
> > servicemix.maximumPoolSize = 256
> > servicemix.queueSize = 256
> >
> > I started to get such exception:
> >
> > 05:06:21,891 | ERROR | pool-2-thread-359 | SimpleScheduler
|
> > duler.simple.SimpleScheduler$4 347 | Error while executing
> transaction
> > org.apache.ode.bpel.iapi.ContextException: The execution queue is
> backed
> > up... Forcing ContextException
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
> > pleScheduler.java:221)
> > at
> >
>
org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
> > java:279)
> > at
> >
>
org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
> > java:364)
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > .java:335)
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > .java:334)
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
> > heduler.java:173)
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > ava:333)
> > at
> >
>
org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > ava:330)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > at
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
> > r.java:650)
> > at
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> > va:675)
> > at java.lang.Thread.run(Thread.java:595)
> >
> >
> > After some minutes application started going to crash (leak memory
and
> > finally OutOfMemoryError).
> >
> > What is the reason ?
> >
> > When I tested 50 parallel clients everything was ok.
> >
> > --
> > Regards
> > Mateusz Nowakowski
> >
>
Re: The execution queue is backed up... Forcing ContextException + MEMORY LEAK ??
Posted by Matthieu Riou <ma...@offthelip.org>.
Hi,
The execution queue doesn't really reflect the number of concurrent
requests, it's size depends more on how busy ODE is at the moment. When ODE
receives requests and starts working on them, it adds things in its "todo"
queue and then picks things up from there. If the global processing is too
slow to consume everything in the todo queue, it will start growing little
by little. The max of 50 just says that you can't keep on piling up work
forever because otherwise you'd just saturate the system completely.
The fact you're seeing this limit shows that you've hit the maximum of
performance for ODE with persistent processes on your current hardware and
with your database configuration. So if you really plan to handle that sort
of traffic in production, you'll have to either switch to in-mem processes,
better tune your database or get better hardware.
For your requests, normally the should time out. But you'll probably want to
check what kind of timeout you have in ServiceMix for requests that can't be
served, this could explain the memory leak (if too many requests are just
there waiting for nothing). I've checked the scheduler and couldn't spot any
possible leak due to this exception being thrown. But some sort of profiling
would help to pinpoint where this comes from.
Hope this helps,
Matthieu
On Dec 4, 2007 3:34 AM, Nowakowski, Mateusz <
Mateusz.Nowakowski@sabre-holdings.com> wrote:
> Yes, I use 120 threads for ode and it is probably maximum considering
> that every service mix components need to have similar own thread pool.
>
>
> I launched longer stress test for 100 clients and for 50 clients. For
> 100 clients application crashed after 3h.
>
> For 50 clients application survives 10h stress test, but gc log indicate
> slow but memory leak. I investigate it, it can be service mix...
>
> but I see the same exception all the time:
>
> 15:02:43,927 | ERROR | pool-10-thread-114 | SimpleScheduler |
> duler.simple.SimpleScheduler$4 353 | Error while executing transaction
> org.apache.ode.bpel.iapi.ContextException: The execution queue is backed
> up... Forcing ContextException
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
> pleScheduler.java:227)
> at
> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
> java:290)
> at
> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
> java:373)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> .java:341)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> .java:340)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
> heduler.java:179)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> ava:339)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> ava:336)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
> r.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:675)
> at java.lang.Thread.run(Thread.java:595)
>
>
> It is impossible to fill ode thread poll due to it is set to 120
> threads, and the maximum parallel requests are 50.
>
> I dig further and see that SimpleScheduler uses hardcoded size of
> concurrent task which is 200. Still maximum concurrent requests is 50.
>
> What is done with the request when this exception is thrown ??
> How to get rid of this ??
>
> Please help.
>
> --
> Regards
> Mateusz Nowakowski
>
> -----Original Message-----
> From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On Behalf
> Of Matthieu Riou
> Sent: Thursday, November 29, 2007 5:21 PM
> To: dev@ode.apache.org
> Subject: Re: The execution queue is backed up... Forcing
> ContextException
>
> Do you saturate your CPU (about 85-90%)? If not, more threads should
> bring
> you more performance at least for in-memory. For persistent processes,
> it's
> the database optimization game...
>
> Cheers,
> Matthieu
>
> On Nov 29, 2007 8:01 AM, Nowakowski, Mateusz <
> Mateusz.Nowakowski@sabre-holdings.com> wrote:
>
> > Thanks. It helps a bit. Application doesn't crash anymore but
> > performance is decreased than for 50 parallel clients... But it can be
> > different reason (not enough RAM or CPU)
> >
> > I know about this property before but I don't use because I assume
> that
> > BPEL caches threads and creates new thread when necessary.
> > I looked on ODEService.initBpelServer() and saw:
> >
> > if (_ode._config.getThreadPoolMaxSize() == 0)
> > _ode._executorService = Executors.newCachedThreadPool();
> > else
> > _ode._executorService =
> > Executors.newFixedThreadPool(_ode._config.getThreadPoolMaxSize());
> >
> >
> > --
> > Regards
> > Mateusz Nowakowski
> >
> > -----Original Message-----
> > From: matthieu.riou@gmail.com [mailto:matthieu.riou@gmail.com] On
> Behalf
> > Of Matthieu Riou
> > Sent: Wednesday, November 28, 2007 5:17 PM
> > To: dev@ode.apache.org
> > Subject: Re: The execution queue is backed up... Forcing
> > ContextException
> >
> > Hi Mateusz,
> >
> > I've guessing you're exhausting ODE's thread pool. When the engine
> can't
> > keep up anymore and messages are coming faster that they can be
> > consumed,
> > the execution queue gets backed up and instead of accumulating
> messages
> > we
> > throw an exception.
> >
> > To increase the size of the pool, you can set the following property
> in
> > ODE's configuration:
> >
> > ode-jbi.threads.pool.size=100
> >
> > Let us know how it goes.
> >
> > Cheers,
> > Matthieu
> >
> > On Nov 28, 2007 6:56 AM, Nowakowski, Mateusz <
> > Mateusz.Nowakowski@sabre-holdings.com> wrote:
> >
> > > I've performing some performance tests using Fuse 3.3.0.2 and Apache
> > ODE
> > > 1.1.
> > >
> > > I tested it using 100 parallel clients.
> > > I used such thread pool properties:
> > > servicemix.corePoolSize = 32
> > > servicemix.maximumPoolSize = 256
> > > servicemix.queueSize = 256
> > >
> > > I started to get such exception:
> > >
> > > 05:06:21,891 | ERROR | pool-2-thread-359 | SimpleScheduler
> |
> > > duler.simple.SimpleScheduler$4 347 | Error while executing
> > transaction
> > > org.apache.ode.bpel.iapi.ContextException: The execution queue is
> > backed
> > > up... Forcing ContextException
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler.schedulePersistedJob(Sim
> > > pleScheduler.java:221)
> > > at
> > >
> >
> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.
> > > java:279)
> > > at
> > >
> >
> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.
> > > java:364)
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > > .java:335)
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(SimpleScheduler
> > > .java:334)
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleSc
> > > heduler.java:173)
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > > ava:333)
> > > at
> > >
> >
> org.apache.ode.scheduler.simple.SimpleScheduler$4.call(SimpleScheduler.j
> > > ava:330)
> > > at
> > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:123)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
> > > r.java:650)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> > > va:675)
> > > at java.lang.Thread.run(Thread.java:595)
> > >
> > >
> > > After some minutes application started going to crash (leak memory
> and
> > > finally OutOfMemoryError).
> > >
> > > What is the reason ?
> > >
> > > When I tested 50 parallel clients everything was ok.
> > >
> > > --
> > > Regards
> > > Mateusz Nowakowski
> > >
> >
>