Performance question when clustering

[Dennis van der Laan <d....@rug.nl>]

Hi all,

We are setting up a clustered Jackrabbit environment as the data storage
for our (custom) CMS. We are using Jackrabbit 1.6.0 with an Oracle 10g
database, bundled persistence manager and finegrained ISM locking.
Whenever the repository is accessed through a JcrSession, we first do a
session.refresh().  We were assuming clustering would have some
overhead, but because of the much-talked-about performance bottleneck in
the PersistenceManager and/or SharedItemStateManager, the performance
boost would compensate this overhead.

What we have observed is that having more cluster nodes using the same
Jackrabbit repository has a significant impact on performance, with
almost linear degradation. We started a performance test by uploading
files to just one machine in a two-machine-cluster and the same test on
a six-machine-cluster. The first test was about 3x faster than the
second test. Most of the time, the threads are waiting on a Mutex in
org.apache.jackrabbit.core.cluster.ClusterNode.sync() (called from
SessionImpl.refresh()). Uploading to multiple cluster-machines at the
same time only seemed to increase the performance-impact, probably
because the time a Mutex is being held is longer due to the fact other
nodes in the cluster have updates, too.

So my questions are:
- Is it a good idea to call session.refresh() every time we use the
session?
- Is there a difference between calling session.refresh() and the
automatic sync done by the ClusterNode thread?
- Why is a refresh more expansive when there are more cluster nodes?

Thanks for any information!

Dennis

-- 
Dennis van der Laan, MSc
Centre for Information Technology
University of Groningen

Re: Performance question when clustering

[Patricio Echagüe <pa...@gmail.com>]

It might be a late reply but we i saw at least in JR 2.0 is that when
session.save() happens it also forces a sync first so that the JR node gets
up-to-date first.

We also see the degradation in performance the more nodes we put in the
cluster. The global lock forces to make every session.save() sequential
since it acquires a global lock.

My 2 cents

On Tue, Apr 20, 2010 at 4:49 AM, Thomas Müller <th...@day.com>wrote:

> Hi,
>
> > automatic sync and the manual Session.refresh() have essentially the
> > same result?
>
> I think so.
>
> > Should this be something you should know?
>
> I don't plan to find out, sorry. I didn't write the cluster code, and
> I don't feel responsible for it :-)
>
> Regards,
> Thomas
>



-- 
Patricio.-

Re: Performance question when clustering

[Thomas Müller <th...@day.com>]

Hi,

> automatic sync and the manual Session.refresh() have essentially the
> same result?

I think so.

> Should this be something you should know?

I don't plan to find out, sorry. I didn't write the cluster code, and
I don't feel responsible for it :-)

Regards,
Thomas

Re: Performance question when clustering

[Dennis van der Laan <d....@rug.nl>]

Thomas, others,
> The automatic sync is periodic (once every few seconds).
> Session.refresh() will force an immediate cluster sync. I'm not sure
> if the immediate cluster sync on Session.refresh() is a good idea, but
> that's how it's implemented currently.
>   
Okay, I agree the choice to do a full cluster sync for every
Session.refresh() can be a poor one in some situations. But I guess
someone would like to avoid having to call ClusterNode.sync() from
custom code if needed, because of code-dependency on a clustered
configuration. A trade-off. But from your reply I may conclude the
automatic sync and the manual Session.refresh() have essentially the
same result?
>> - Why is a refresh more expansive when there are more cluster nodes?
>>     
> This I don't know.
>   
I meant 'expensive' and not 'expansive' :) But I guess you guessed that.
Should this be something you should know? Is it strange behavior? Or
might somebody else know why we see this effect?

Thanks,

Dennis

-- 
Dennis van der Laan, MSc
Centre for Information Technology
University of Groningen

Re: Performance question when clustering

[Thomas Müller <th...@day.com>]

Hi,

The automatic sync is periodic (once every few seconds).
Session.refresh() will force an immediate cluster sync. I'm not sure
if the immediate cluster sync on Session.refresh() is a good idea, but
that's how it's implemented currently.

> - Why is a refresh more expansive when there are more cluster nodes?

This I don't know.

Regards,
Thomas