You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ode.apache.org by arunjolly <jo...@gmail.com> on 2013/01/24 17:22:51 UTC

Apache ODE 1.3.5 - In Memory Execution - How stable ?

Hi,

We use apache ode 1.3.3 inside Servicemix 3.3.2 for implementing our
workflows. When the load increases (ie number of flow invocations per unit
time ), ode freezes, stops processing new requests and hangs. Usullay, we
solve the issue by doing a Servicemix restart after a cleanup of the 'data'
folder.

Initially, i thought that this was a thread problem ( lack of threads for
ode to use ). After increasing the 'thread pool size' in ode-jbi.properties,
this problem got addressed to an extend. But still, under heavy load, ode
continues to hang.

Further investigation, with additional logging revealed that under heavy
load, ode is unable to get enough connections to the DB
(NoManagedConnectionException ) to persist its state. Our ode persistence
configuration is INTERNAL (refer to ode-jbi.db.mode=INTERNAL ) where the
connection pool is maintained by the container/runtime itself. Efforts to
migrate it to a EXTERNAL jndi datasource configuration ( where we can
configure the connection pool parameters ) has not yet borne fruit ( Thanks
to my inability to comprehend the zillion errors that keep cropping up at
each stage :( :) )

So i tried the other available option, 'In-Memory execution' of my
processes. This works fine in a 'test setup' and it no longer throws the
'NoManageConnectionException' under simulated load.

But i have a few concerns about moving this to PROD.  How different is
In-Memory execution different from 'persisted' execution ? How much more
memory will this consume ? Will this affect the reliability of my PROD setup
by causing 'OutOfMemoryOutages'.

We have about 10 deployed bpel processes. And the load (no. of requests
)...well it is usually minimal, but sometimes spikes up ( that's where i
need my processes to run really fast but without causing any OutOfMemory
PROD outages...hopefully :D :) :P )

Need your thoughts, suggestions and advice on this.

Thanks in advance,
Arun
-- 
View this message in context: http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941139.html
Sent from the Apache Ode User mailing list archive at Nabble.com.

Re: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by arunjolly <jo...@gmail.com>.

Abdul,

Let me try to explain better...

It's not actually a memory leak that is causing the issue. When i configure
the ode in INTERNAL mode with a thread pool configuration of 40, under heavy
load, the ode easily runs out of DB connections resulting in
'NoManageConnections Available' Exception.

To counter this, i've now made changes to my deploy.xml configuration to
execute the bpel processes in memory without any persistence. This works
fine under simulated load in the TEST setup that i've created.

But im not really sure about the scalability ( Memory consumption wise ) and
the reliability of this model, need some thoughts on this cos i dont want to
take chances with my PROD setup.

Thanks,
Arun
-- 
View this message in context: http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941889.html
Sent from the Apache Ode User mailing list archive at Nabble.com.

Re: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by abdulkadir yaman <ab...@gmail.com>.
Dear Arun,

Have you had a chance to take a snapshot of the memory while ode freezes
before restarting servicemix? If you investigate the memory dump you will
going to have a chance to find the leak.

regards


On Thu, Jan 24, 2013 at 6:22 PM, arunjolly <jo...@gmail.com> wrote:

>
> Hi,
>
> We use apache ode 1.3.3 inside Servicemix 3.3.2 for implementing our
> workflows. When the load increases (ie number of flow invocations per unit
> time ), ode freezes, stops processing new requests and hangs. Usullay, we
> solve the issue by doing a Servicemix restart after a cleanup of the 'data'
> folder.
>
> Initially, i thought that this was a thread problem ( lack of threads for
> ode to use ). After increasing the 'thread pool size' in
> ode-jbi.properties,
> this problem got addressed to an extend. But still, under heavy load, ode
> continues to hang.
>
> Further investigation, with additional logging revealed that under heavy
> load, ode is unable to get enough connections to the DB
> (NoManagedConnectionException ) to persist its state. Our ode persistence
> configuration is INTERNAL (refer to ode-jbi.db.mode=INTERNAL ) where the
> connection pool is maintained by the container/runtime itself. Efforts to
> migrate it to a EXTERNAL jndi datasource configuration ( where we can
> configure the connection pool parameters ) has not yet borne fruit ( Thanks
> to my inability to comprehend the zillion errors that keep cropping up at
> each stage :( :) )
>
> So i tried the other available option, 'In-Memory execution' of my
> processes. This works fine in a 'test setup' and it no longer throws the
> 'NoManageConnectionException' under simulated load.
>
> But i have a few concerns about moving this to PROD.  How different is
> In-Memory execution different from 'persisted' execution ? How much more
> memory will this consume ? Will this affect the reliability of my PROD
> setup
> by causing 'OutOfMemoryOutages'.
>
> We have about 10 deployed bpel processes. And the load (no. of requests
> )...well it is usually minimal, but sometimes spikes up ( that's where i
> need my processes to run really fast but without causing any OutOfMemory
> PROD outages...hopefully :D :) :P )
>
> Need your thoughts, suggestions and advice on this.
>
> Thanks in advance,
> Arun
> --
> View this message in context:
> http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941139.html
> Sent from the Apache Ode User mailing list archive at Nabble.com.
>
>

Re: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by Lorenzo Bolzani <l....@gmail.com>.
Hi Arun,
we used some in-memory processes for a while in production (servicemix
3.4.0/ode 1.3.5). It works, but there are some issues.

The main one is the limit with the messages queue. Under heavy load the
engine starts logging

Discarding in-memory instance 11149 because it exceeded its time-to-live:
mem.instance(type={it.something.
xyz}outbound-sendpec-process iid=11149)

and new "fresh" requests are not processed. See:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.ode/ode-bpel-runtime/1.3.5/org/apache/ode/bpel/memdao/ProcessDaoImpl.java#ProcessDaoImpl.discardOldInstances%28%29

This should happen only for very old messages, but somehow the engine
stopped processing all messages as soon as this error appeared. Restarting
the instance was not enough (?) and we had to undeploy the process and
clean everything to resume the operations. I did not do further
investigations.

We slowed down the producer process and now it works fine, but it think
we'll move this process back to database mode.


Bye

Lorenzo

Re: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by Tammo van Lessen <tv...@gmail.com>.
Hi Arun,

also, you better not use Derby. It is packaged with ODE to provide a hassle
free quickstart, however it is not really suitable for a use in production.
I'd rather try MySQL which works really well or any other MVCC-enabled
database. If you prefer in-mem-DBs, you may give H2 a try.

The in-mem-mode of ODE imposes a bunch of restrictions on the process
execution. IIRC, processes can not have more than one receive activity,
correlation is not fully supported and the process management API cannot be
used for queries. I'd try with a different engine first.

HTH,
  Tammo


On Fri, Jan 25, 2013 at 7:48 AM, Nowakowski, Mateusz <
Mateusz.Nowakowski@sabre.com> wrote:

> You wrote that use ode 1.3.3 or ode 1.3.5?
> ODE 1.3.4 or 1.3.3 (I don't remember which one) had a general performance
> issue which was solved definitely in ode 1.3.5.
>
> Secondly use database in-memory mode as it is a bootleneck for the high
> speed in-memory processes.
>
> Add to jndi.xml entry:
>         <entry key="java:comp/env/jdbc/ode">
>         <bean class="org.apache.derby.jdbc.EmbeddedDataSource">
>                 <property name="databaseName" value="memory:ODEDB"/>
>                 <property name="connectionAttributes"
> value=";restoreFrom=data/smx/components/OdeBpelEngine/version_1/jpadb"/>
>                 <property name="user" value="sa"/>
>         </bean>
>         </entry>
>
> And change ode.properties:
>
> ode-jbi.db.mode=EXTERNAL
> ode-jbi.db.ext.dataSource=java:comp/env/jdbc/ode
>
>
> Then you can try to tweek other properties:
>
> ode-jbi.threads.pool.size=120
>
> # To mitigate ODE-943
> ode.scheduler.queueLength=2147483647
> ode.scheduler.transactionsPerSecond=1000
>
>
> --
> Regards,
> Mateusz Nowakowski
>
> -----Original Message-----
> From: arunjolly [mailto:jollyarun@gmail.com]
> Sent: Thursday, January 24, 2013 17:23
> To: user@ode.apache.org
> Subject: Apache ODE 1.3.5 - In Memory Execution - How stable ?
>
>
> Hi,
>
> We use apache ode 1.3.3 inside Servicemix 3.3.2 for implementing our
> workflows. When the load increases (ie number of flow invocations per unit
> time ), ode freezes, stops processing new requests and hangs. Usullay, we
> solve the issue by doing a Servicemix restart after a cleanup of the 'data'
> folder.
>
> Initially, i thought that this was a thread problem ( lack of threads for
> ode to use ). After increasing the 'thread pool size' in
> ode-jbi.properties, this problem got addressed to an extend. But still,
> under heavy load, ode continues to hang.
>
> Further investigation, with additional logging revealed that under heavy
> load, ode is unable to get enough connections to the DB
> (NoManagedConnectionException ) to persist its state. Our ode persistence
> configuration is INTERNAL (refer to ode-jbi.db.mode=INTERNAL ) where the
> connection pool is maintained by the container/runtime itself. Efforts to
> migrate it to a EXTERNAL jndi datasource configuration ( where we can
> configure the connection pool parameters ) has not yet borne fruit ( Thanks
> to my inability to comprehend the zillion errors that keep cropping up at
> each stage :( :) )
>
> So i tried the other available option, 'In-Memory execution' of my
> processes. This works fine in a 'test setup' and it no longer throws the
> 'NoManageConnectionException' under simulated load.
>
> But i have a few concerns about moving this to PROD.  How different is
> In-Memory execution different from 'persisted' execution ? How much more
> memory will this consume ? Will this affect the reliability of my PROD
> setup by causing 'OutOfMemoryOutages'.
>
> We have about 10 deployed bpel processes. And the load (no. of requests
> )...well it is usually minimal, but sometimes spikes up ( that's where i
> need my processes to run really fast but without causing any OutOfMemory
> PROD outages...hopefully :D :) :P )
>
> Need your thoughts, suggestions and advice on this.
>
> Thanks in advance,
> Arun
> --
> View this message in context:
> http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941139.html
> Sent from the Apache Ode User mailing list archive at Nabble.com.
>
>


-- 
Tammo van Lessen - http://www.taval.de

RE: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by "Nowakowski, Mateusz" <Ma...@sabre.com>.
You wrote that use ode 1.3.3 or ode 1.3.5?
ODE 1.3.4 or 1.3.3 (I don't remember which one) had a general performance issue which was solved definitely in ode 1.3.5.

Secondly use database in-memory mode as it is a bootleneck for the high speed in-memory processes.

Add to jndi.xml entry:
	<entry key="java:comp/env/jdbc/ode">
	<bean class="org.apache.derby.jdbc.EmbeddedDataSource">
		<property name="databaseName" value="memory:ODEDB"/>
		<property name="connectionAttributes" value=";restoreFrom=data/smx/components/OdeBpelEngine/version_1/jpadb"/>	
		<property name="user" value="sa"/>
	</bean>
	</entry>

And change ode.properties:

ode-jbi.db.mode=EXTERNAL
ode-jbi.db.ext.dataSource=java:comp/env/jdbc/ode


Then you can try to tweek other properties:

ode-jbi.threads.pool.size=120

# To mitigate ODE-943
ode.scheduler.queueLength=2147483647
ode.scheduler.transactionsPerSecond=1000


-- 
Regards,
Mateusz Nowakowski

-----Original Message-----
From: arunjolly [mailto:jollyarun@gmail.com] 
Sent: Thursday, January 24, 2013 17:23
To: user@ode.apache.org
Subject: Apache ODE 1.3.5 - In Memory Execution - How stable ?


Hi,

We use apache ode 1.3.3 inside Servicemix 3.3.2 for implementing our workflows. When the load increases (ie number of flow invocations per unit time ), ode freezes, stops processing new requests and hangs. Usullay, we solve the issue by doing a Servicemix restart after a cleanup of the 'data'
folder.

Initially, i thought that this was a thread problem ( lack of threads for ode to use ). After increasing the 'thread pool size' in ode-jbi.properties, this problem got addressed to an extend. But still, under heavy load, ode continues to hang.

Further investigation, with additional logging revealed that under heavy load, ode is unable to get enough connections to the DB (NoManagedConnectionException ) to persist its state. Our ode persistence configuration is INTERNAL (refer to ode-jbi.db.mode=INTERNAL ) where the connection pool is maintained by the container/runtime itself. Efforts to migrate it to a EXTERNAL jndi datasource configuration ( where we can configure the connection pool parameters ) has not yet borne fruit ( Thanks to my inability to comprehend the zillion errors that keep cropping up at each stage :( :) )

So i tried the other available option, 'In-Memory execution' of my processes. This works fine in a 'test setup' and it no longer throws the 'NoManageConnectionException' under simulated load.

But i have a few concerns about moving this to PROD.  How different is In-Memory execution different from 'persisted' execution ? How much more memory will this consume ? Will this affect the reliability of my PROD setup by causing 'OutOfMemoryOutages'.

We have about 10 deployed bpel processes. And the load (no. of requests )...well it is usually minimal, but sometimes spikes up ( that's where i need my processes to run really fast but without causing any OutOfMemory PROD outages...hopefully :D :) :P )

Need your thoughts, suggestions and advice on this.

Thanks in advance,
Arun
--
View this message in context: http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941139.html
Sent from the Apache Ode User mailing list archive at Nabble.com.

RE: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by arunjolly <jo...@gmail.com>.

Hmm...Thanks Frank

Found the following properties for the ode-jbi distribution.

ode-jbi.db.pool.max
ode-jbi.db.pool.min

Although, not sure why the same is not documented in the official
documentation. Will try this and let you know what happens.

Meanwhile...my thread pool size is 40, what do you think would be an
appropriate max value for the db pool so that the number of connections
don't get exhausted by 40 concurrently running threads ? Ideas ?

Thanks,
Arun
-- 
View this message in context: http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941862.html
Sent from the Apache Ode User mailing list archive at Nabble.com.

RE: Apache ODE 1.3.5 - In Memory Execution - How stable ?

Posted by "Frank, Tim" <ti...@siemens.com>.
We have a slightly different setup... however we found the setting:

ode-axis2.db.pool.max

defaults to 10....

So bumping this up allowed for more simultaneous execution.



-----Original Message-----
From: arunjolly [mailto:jollyarun@gmail.com] 
Sent: Thursday, January 24, 2013 10:23 AM
To: user@ode.apache.org
Subject: Apache ODE 1.3.5 - In Memory Execution - How stable ?


Hi,

We use apache ode 1.3.3 inside Servicemix 3.3.2 for implementing our workflows. When the load increases (ie number of flow invocations per unit time ), ode freezes, stops processing new requests and hangs. Usullay, we solve the issue by doing a Servicemix restart after a cleanup of the 'data'
folder.

Initially, i thought that this was a thread problem ( lack of threads for ode to use ). After increasing the 'thread pool size' in ode-jbi.properties, this problem got addressed to an extend. But still, under heavy load, ode continues to hang.

Further investigation, with additional logging revealed that under heavy load, ode is unable to get enough connections to the DB (NoManagedConnectionException ) to persist its state. Our ode persistence configuration is INTERNAL (refer to ode-jbi.db.mode=INTERNAL ) where the connection pool is maintained by the container/runtime itself. Efforts to migrate it to a EXTERNAL jndi datasource configuration ( where we can configure the connection pool parameters ) has not yet borne fruit ( Thanks to my inability to comprehend the zillion errors that keep cropping up at each stage :( :) )

So i tried the other available option, 'In-Memory execution' of my processes. This works fine in a 'test setup' and it no longer throws the 'NoManageConnectionException' under simulated load.

But i have a few concerns about moving this to PROD.  How different is In-Memory execution different from 'persisted' execution ? How much more memory will this consume ? Will this affect the reliability of my PROD setup by causing 'OutOfMemoryOutages'.

We have about 10 deployed bpel processes. And the load (no. of requests )...well it is usually minimal, but sometimes spikes up ( that's where i need my processes to run really fast but without causing any OutOfMemory PROD outages...hopefully :D :) :P )

Need your thoughts, suggestions and advice on this.

Thanks in advance,
Arun
--
View this message in context: http://old.nabble.com/Apache-ODE-1.3.5---In-Memory-Execution---How-stable---tp34941139p34941139.html
Sent from the Apache Ode User mailing list archive at Nabble.com.