org/apache/ode/scheduler/simple/SimpleScheduler.java_processedSinceLastLoadTask concurrent map keeps growing under load test

[abdulkadir yaman <ab...@gmail.com>]

Hi,

I deployed ode 1.3.5 in servicemix 4.3.0 on Ubuntu 11.04 64bit with maxHeap
size = 1024m . I have set necessary coreThreadPool sizes to 16,
maxThreadPool sizes to unlimited. I generated a 300-400 tps load on a flow
which is in-memory enabled. After a while, short gc cycles could not achieve
to collect necessary objects and fullgc cycles try to collect with pausing
system 2,5 - 4.0 seconds which causes system to freeze and not handle
further requests. At the same time i took a heap snapshot and tried to
analyse it. As a result, i saw the biggest object was SimpleScheduler
instance containing ConcurrentHashMap<String,Long>
_processedSinceLastLoadTask instance (almost 1gb) which contains
jobId(String) type as keys and scheduling dates(Long) as values.

I have researched a bit on SimpleScheduler trunk version but seen that 1.3.5
version installed in Servicemix 4.3.0 contains an older version compared to
trunk. I decompiled the class in servicemix version and investigated that
either _processedSinceLastLoadTask.clear() does not work or execution does
not endure till the line  _processedSinceLastLoadTask.clear() inside
 boolean doLoadImmediate() method block.

I have tried several configuration parameters such as `process-events
generate="none"` `cleanup on="always"` in deploy.xml
or ode-jbi.threads.pool.size=100 in ode config file. None of them have
worked out.

Any idea about what may cause _processedSinceLastLoadTask map to grow?

Thanks.