Problem importing more than 5K files over WebDAV

[Brian Topping <to...@codehaus.org>]

Greetings all,

I'm working to get my first Jackrabbit repository deployed and am  
having some pretty serious problems loading it via WebDAV.  It's  
generally working pretty well, but due to some brain damaged legacy, I  
have a directory with about 42K objects in the top-level directory and  
about 80K objects in the rest of the more evenly loaded tree.

The problem that I am seeing is that after importing a few thousand  
files, the speed that the import proceeds starts to slow  
exponentially.  Log messages start being generated from LRUNodeIdCache  
and BundleCache more rapidly, and intake that was running at about  
twenty files per second (and maybe one log message per 500 files)  
start slowing to one file every few seconds and the above log messages  
repeating multiple times per file.  I generally don't wait much longer  
than this, there are 73,000+ files left to go!

In attempts to resolve this, I've tried both the  
DerbyPersistenceManager and BundleFsPersistenceManager to no avail.   
Performance seems to improve if the SearchIndex is left unconfigured,  
but only to the extent that maybe 10% more items are imported before  
the performance falls off.

What I'd like to find from the experienced users here it is practical  
under any circumstances to have this many files in a directory, and if  
so, what I might tune to help the import along.

Kind regards,

Brian

Re: Problem importing more than 5K files over WebDAV

[Brian Topping <to...@codehaus.org>]

Hi Marcel,

Thanks for your response and interest.  I did increase the JVM, but it  
turns out that it was how I was connecting... as OS-X WebDAV FS saves  
a file, a separate thread seems to ask for a lot of information to  
update the directory view or something.  Since there were so many  
nodes in the directory, it was a very expensive operation and got more  
so with each new file.

Writing an importer that talks directly to the JCR APIs did the trick.

Something good to take note of, I guess.

Cheers, Brian

On Apr 15, 2009, at 8:16 AM, Marcel Reutegger wrote:

> Hi,
>
> On Tue, Apr 14, 2009 at 20:44, Brian Topping <to...@codehaus.org>  
> wrote:
>> In attempts to resolve this, I've tried both the  
>> DerbyPersistenceManager and
>> BundleFsPersistenceManager to no avail.  Performance seems to  
>> improve if the
>> SearchIndex is left unconfigured, but only to the extent that maybe  
>> 10% more
>> items are imported before the performance falls off.
>
> did you try to increase the heap size for the JVM already? Does it
> have an effect on when the import performance drops significantly?
>
> regards
> marcel
>

Re: Problem importing more than 5K files over WebDAV

[Marcel Reutegger <ma...@gmx.net>]

Hi,

On Tue, Apr 14, 2009 at 20:44, Brian Topping <to...@codehaus.org> wrote:
> In attempts to resolve this, I've tried both the DerbyPersistenceManager and
> BundleFsPersistenceManager to no avail.  Performance seems to improve if the
> SearchIndex is left unconfigured, but only to the extent that maybe 10% more
> items are imported before the performance falls off.

did you try to increase the heap size for the JVM already? Does it
have an effect on when the import performance drops significantly?

regards
 marcel