Jackrabbit In Low Memory Situations

[Adrian Sutton <ad...@symphonious.net>]

Hi all,
We've developed a web app that's using Jackrabbit as it's content  
repository and need to deploy it into an environment with quite  
limited memory and are seeing a couple of problems:

* When adding a large amount of data to the repository (via a series  
of write operations, not a JCR import as the existing data isn't in a  
JCR repository), we encounter an out of memory exception. If we do  
the import with an increased amount of RAM (128M) the import goes  
through correctly and we can then restart and run the web app in 64M  
of RAM without further problems.

* When running in limited RAM, we're finding the web app runs quite  
slowly - more so than any other web apps (with very similar  
functionality) on the same server.

We obviously will need to do a fair bit of profiling work and  
determine particular bottlenecks etc, however with the limited  
knowledge of Jackrabbit that we have at this stage, it's a difficult  
task. Are there particular configuration options we should  
investigate that may help? At the moment we're using the default  
configuration which is unlikely to be ideal.

Vague pointers and general hand waving type answers are welcome (as  
are silver bullets but they're harder to come by).

Thanks in advance,

Adrian Sutton
http://www.symphonious.net

Re: Jackrabbit In Low Memory Situations

[Adrian Sutton <ad...@symphonious.net>]

Hi Thomas and Felix,
Thanks for your help, this gives me a few things to go on.

> - Transient changes / large transactions. Jackrabbit currently keeps
> unsaved changes in memory. I suggest you call Session.save from time
> to time in a large import.

We were calling session.save after every page (which is slowing  
things down, but not solving the problem outright). We weren't  
logging out and back in to create a new session though. Either way,  
the import isn't a problem as we can do it on a system with more  
memory available and it's only a once off operation anyway.

> - Caches from persistence managers (Derby uses a lot of memory)

This sounds like a promising lead - we want to evaluate switching to  
using MySQL instead of Derby since we already have the MySQL server  
anyway and we have the backup procedures in place for them. Sounds  
like it has potential to reduce our memory woes as well.

This gives me a bunch of other configuration options that I can tweak  
to get an idea of their effects as well which is very helpful. I've  
also managed to get some rough profiling results and tweaked things  
to pull less information out of the repository for each request which  
has gotten things to a reasonable point. With a few more tweaks in  
the right places it should be fast enough.

Thanks for the help.

Regards,

Adrian Sutton
http://www.symphonious.net



On 26/06/2007, at 4:52 PM, Thomas Mueller wrote:

> Hi,
>
> I don't think the Jackrabbit internal cache is the problem in your
> case. Usually the main reasons for memory problems are (as Felix
> wrote):
>
> - Transient changes / large transactions. Jackrabbit currently keeps
> unsaved changes in memory. I suggest you call Session.save from time
> to time in a large import.
>
> - Caches from persistence managers (Derby uses a lot of memory)
>
> To find out what component uses the most memory, I suggest you use a
> tool such as the YourKit Java Pofiler (free trial and open source
> licenses available).
>
> Jackrabbit internal caching: Yes, this is configurable. There was a
> patch for Jackrabbit 1.3, see
> http://issues.apache.org/jira/browse/JCR-725. The configuration is
> Jackrabbit specific:
>
> CacheManager manager = ((RepositoryImpl) repository).getCacheManager 
> ();
> manager.setMaxMemory(8 * 1024 * 1024); // default is 16 * 1024 * 1024
> manager.setMaxMemoryPerCache(1024 * 1024); // default is 4 * 1024 *  
> 1024
> manager.setMinMemoryPerCache(64 * 1024); // default is 128 * 1024
>
> The default values should be OK for most situations.
>
> Thomas

Re: Jackrabbit In Low Memory Situations

[Thomas Mueller <th...@gmail.com>]

Hi,

I don't think the Jackrabbit internal cache is the problem in your
case. Usually the main reasons for memory problems are (as Felix
wrote):

- Transient changes / large transactions. Jackrabbit currently keeps
unsaved changes in memory. I suggest you call Session.save from time
to time in a large import.

- Caches from persistence managers (Derby uses a lot of memory)

To find out what component uses the most memory, I suggest you use a
tool such as the YourKit Java Pofiler (free trial and open source
licenses available).

Jackrabbit internal caching: Yes, this is configurable. There was a
patch for Jackrabbit 1.3, see
http://issues.apache.org/jira/browse/JCR-725. The configuration is
Jackrabbit specific:

CacheManager manager = ((RepositoryImpl) repository).getCacheManager();
manager.setMaxMemory(8 * 1024 * 1024); // default is 16 * 1024 * 1024
manager.setMaxMemoryPerCache(1024 * 1024); // default is 4 * 1024 * 1024
manager.setMinMemoryPerCache(64 * 1024); // default is 128 * 1024

The default values should be OK for most situations.

Thomas

Re: Jackrabbit In Low Memory Situations

[Felix Meschberger <Fe...@day.com>]

Hi Adrian,

The first situation is due to the implementation of transient items (items
created, modfied, deleted but not saved yet) and has been discussed multiple
times on this list. You can help yourself by importing the data in smaller
junks to limit the number of transient items.

The second situation is probably somewhat harder to come by. For one, it
depends on the persistence manager you are using. If you happen to use e.g.
derby or hsql some memory is used by their internal caching. So you will
have to check the respective persistence manager configuration to limit that
caching. BTW: this caching is not of a big use anyway, as Jackrabbit has
internal caching, which is more likely to match the use case of the
Jackrabbit application.

In addition as I said, Jackrabbit has internal caching, which is probably -
not sure actually, but I think I remember having read something like that -
also configurable to a certain extent. Maybe Stephan or Thomas Müller might
jump in here.

Hope this helps a bit

Regards
Felix

On 6/26/07, Adrian Sutton <ad...@symphonious.net> wrote:
>
> Hi all,
> We've developed a web app that's using Jackrabbit as it's content
> repository and need to deploy it into an environment with quite
> limited memory and are seeing a couple of problems:
>
> * When adding a large amount of data to the repository (via a series
> of write operations, not a JCR import as the existing data isn't in a
> JCR repository), we encounter an out of memory exception. If we do
> the import with an increased amount of RAM (128M) the import goes
> through correctly and we can then restart and run the web app in 64M
> of RAM without further problems.
>
> * When running in limited RAM, we're finding the web app runs quite
> slowly - more so than any other web apps (with very similar
> functionality) on the same server.
>
> We obviously will need to do a fair bit of profiling work and
> determine particular bottlenecks etc, however with the limited
> knowledge of Jackrabbit that we have at this stage, it's a difficult
> task. Are there particular configuration options we should
> investigate that may help? At the moment we're using the default
> configuration which is unlikely to be ideal.
>
> Vague pointers and general hand waving type answers are welcome (as
> are silver bullets but they're harder to come by).
>
> Thanks in advance,
>
> Adrian Sutton
> http://www.symphonious.net
>
>
>
>