You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Jerry Lacy <je...@ssglimited.com> on 2007/01/25 21:56:13 UTC
Threading issues
Hello,
I was hoping someone could shed some light on some threading issues we are
experiencing. We have developed an application using JBoss Portal 2.4 which
uses JackRabbit 1.0 as its content management repository. We recently started
doing load testing and have run into two different threading/deadlock issues
within the JackRabbit code. I’m not sure which side of the JBoss/JackRabbit
fence the issue is on but I thought I would start here… sorry to be so
verbose….
Here is basically what happens in JBoss Portal code for each CMS request (we
are using JAAS):
org.apache.jackrabbit.core.XASession session =
(org.apache.jackrabbit.core.XASession) jcr.login();
XAResource xares = session.getXAResource();
Xid xid = new Xid()
{
public byte[] getBranchQualifier()
{
return new byte[0];
}
public int getFormatId()
{
return 0;
}
public byte[] getGlobalTransactionId()
{
return new byte[0];
}
};
boolean shouldCommit = false;
try {
xares.start(xid, XAResource.TMNOFLAGS);
…
Node node = (Node) session.getItem (…);
…
xares.end(xid, XAResource.TMSUCCESS);
xares.prepare(xid);
shouldCommit = true;
} catch(Exception e) {
...
}
finally {
if (session != null) {
try {
session.save();
if(shouldCommit) {
xares.commit(xid, false);
} else {
xares.rollback(xid);
}
session.logout();
}
catch(Exception e) {
....
}
}
}
One of our scenarios involves a large number of concurrent users with each
http request causing multiple reads through the above code. In this scenario
there is not a JTA transaction in effect. What happens is that we end up with
a large number of Tomcat threads stuck at the following spot (jconsole output):
Name: http-0.0.0.0-8080-19
State: RUNNABLE
Total blocked: 32,519 Total waited: 127
Stack trace:
java.util.HashMap.get(HashMap.java:329)
org.apache.jackrabbit.core.XASessionImpl.start(XASessionImpl.java:231)
…
Each of these threads is churning endlessly and the CPU ends up pegged.
The second scenario is the same as the first except that there is JTA
transaction in effect. The results here are different in that the app will
run well for a while until a deadlock suddenly occurs and everything
immediately grinds to a halt. We consistently end up with two threads
contending for a read and a write lock on the SharedItemStateManager:
Name: http-0.0.0.0-8080-134
State: WAITING on
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@c217c
8
Total blocked: 2,111 Total waited: 150
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acqui
re(WriterPreferenceReadWriteLock.java:163)
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock
(SharedItemStateManager.java:1135)
org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState
(SharedItemStateManager.java:196)
org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState
(LocalItemStateManager.java:86)
org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState
(LocalItemStateManager.java:141)
org.apache.jackrabbit.core.state.XAItemStateManager.getItemState
(XAItemStateManager.java:235)
org.apache.jackrabbit.core.version.XAVersionManager.<init>
(XAVersionManager.java:103)
org.apache.jackrabbit.core.XASessionImpl.createVersionManager
(XASessionImpl.java:152)
org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:247)
org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:208)
org.apache.jackrabbit.core.XASessionImpl.<init>(XASessionImpl.java:97)
org.apache.jackrabbit.core.RepositoryImpl.createSessionInstance
(RepositoryImpl.java:1202)
org.apache.jackrabbit.core.RepositoryImpl.createSession
(RepositoryImpl.java:792)
org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1090)
…
Name: http-0.0.0.0-8080-149
State: WAITING on
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock@17389
c0
Total blocked: 1,879 Total waited: 122
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acqui
re(WriterPreferenceReadWriteLock.java:240)
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireWriteLock
(SharedItemStateManager.java:1148)
org.apache.jackrabbit.core.state.SharedItemStateManager.access$200
(SharedItemStateManager.java:110)
org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin
(SharedItemStateManager.java:461)
org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate
(SharedItemStateManager.java:662)
org.apache.jackrabbit.core.state.XAItemStateManager.prepare
(XAItemStateManager.java:150)
org.apache.jackrabbit.core.TransactionContext.prepare
(TransactionContext.java:128)
org.apache.jackrabbit.core.XASessionImpl.prepare(XASessionImpl.java:300)
…
As a result of the RepositoryImpl.createSession() method being synchronize,
all of the other threads block on RepositoryImpl:
Name: http-0.0.0.0-8080-213
State: BLOCKED on org.apache.jackrabbit.core.RepositoryImpl@287ca7 owned by:
http-0.0.0.0-8080-134
Total blocked: 1,526 Total waited: 5
Stack trace:
org.apache.jackrabbit.core.RepositoryImpl.createSession(RepositoryImpl.java)
org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1090)
…
Is there something we are doing wrong here?
Thanks in advance,
Jerry Lacy
Re: Threading issues
Posted by Tobias Bocanegra <to...@day.com>.
hi,
jackrabbit 1.0 used to have several concurrency issues. i suggest to
upgrade to jacrkabbit 1.2.1 and rerun your tests.
regards, toby
On 1/25/07, Jerry Lacy <je...@ssglimited.com> wrote:
> Hello,
>
> I was hoping someone could shed some light on some threading issues we are
> experiencing. We have developed an application using JBoss Portal 2.4 which
> uses JackRabbit 1.0 as its content management repository. We recently started
> doing load testing and have run into two different threading/deadlock issues
> within the JackRabbit code. I'm not sure which side of the JBoss/JackRabbit
> fence the issue is on but I thought I would start here… sorry to be so
> verbose….
>
> Here is basically what happens in JBoss Portal code for each CMS request (we
> are using JAAS):
>
> org.apache.jackrabbit.core.XASession session =
> (org.apache.jackrabbit.core.XASession) jcr.login();
> XAResource xares = session.getXAResource();
> Xid xid = new Xid()
> {
> public byte[] getBranchQualifier()
> {
> return new byte[0];
> }
> public int getFormatId()
> {
> return 0;
> }
> public byte[] getGlobalTransactionId()
> {
> return new byte[0];
> }
> };
>
> boolean shouldCommit = false;
> try {
> xares.start(xid, XAResource.TMNOFLAGS);
>
> …
> Node node = (Node) session.getItem (…);
> …
>
> xares.end(xid, XAResource.TMSUCCESS);
> xares.prepare(xid);
> shouldCommit = true;
> } catch(Exception e) {
> ...
> }
> finally {
> if (session != null) {
> try {
> session.save();
> if(shouldCommit) {
> xares.commit(xid, false);
> } else {
> xares.rollback(xid);
> }
> session.logout();
> }
> catch(Exception e) {
> ....
> }
> }
> }
>
>
> One of our scenarios involves a large number of concurrent users with each
> http request causing multiple reads through the above code. In this scenario
> there is not a JTA transaction in effect. What happens is that we end up with
> a large number of Tomcat threads stuck at the following spot (jconsole output):
>
> Name: http-0.0.0.0-8080-19
> State: RUNNABLE
> Total blocked: 32,519 Total waited: 127
>
> Stack trace:
> java.util.HashMap.get(HashMap.java:329)
> org.apache.jackrabbit.core.XASessionImpl.start(XASessionImpl.java:231)
> …
>
> Each of these threads is churning endlessly and the CPU ends up pegged.
>
>
> The second scenario is the same as the first except that there is JTA
> transaction in effect. The results here are different in that the app will
> run well for a while until a deadlock suddenly occurs and everything
> immediately grinds to a halt. We consistently end up with two threads
> contending for a read and a write lock on the SharedItemStateManager:
>
> Name: http-0.0.0.0-8080-134
> State: WAITING on
> EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@c217c
> 8
> Total blocked: 2,111 Total waited: 150
>
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:474)
> EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acqui
> re(WriterPreferenceReadWriteLock.java:163)
> org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock
> (SharedItemStateManager.java:1135)
> org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState
> (SharedItemStateManager.java:196)
> org.apache.jackrabbit.core.state.LocalItemStateManager.getNodeState
> (LocalItemStateManager.java:86)
> org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState
> (LocalItemStateManager.java:141)
> org.apache.jackrabbit.core.state.XAItemStateManager.getItemState
> (XAItemStateManager.java:235)
> org.apache.jackrabbit.core.version.XAVersionManager.<init>
> (XAVersionManager.java:103)
> org.apache.jackrabbit.core.XASessionImpl.createVersionManager
> (XASessionImpl.java:152)
> org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:247)
> org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:208)
> org.apache.jackrabbit.core.XASessionImpl.<init>(XASessionImpl.java:97)
> org.apache.jackrabbit.core.RepositoryImpl.createSessionInstance
> (RepositoryImpl.java:1202)
> org.apache.jackrabbit.core.RepositoryImpl.createSession
> (RepositoryImpl.java:792)
> org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1090)
> …
>
> Name: http-0.0.0.0-8080-149
> State: WAITING on
> EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock@17389
> c0
> Total blocked: 1,879 Total waited: 122
>
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:474)
> EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acqui
> re(WriterPreferenceReadWriteLock.java:240)
> org.apache.jackrabbit.core.state.SharedItemStateManager.acquireWriteLock
> (SharedItemStateManager.java:1148)
> org.apache.jackrabbit.core.state.SharedItemStateManager.access$200
> (SharedItemStateManager.java:110)
> org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin
> (SharedItemStateManager.java:461)
> org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate
> (SharedItemStateManager.java:662)
> org.apache.jackrabbit.core.state.XAItemStateManager.prepare
> (XAItemStateManager.java:150)
> org.apache.jackrabbit.core.TransactionContext.prepare
> (TransactionContext.java:128)
> org.apache.jackrabbit.core.XASessionImpl.prepare(XASessionImpl.java:300)
> …
>
> As a result of the RepositoryImpl.createSession() method being synchronize,
> all of the other threads block on RepositoryImpl:
>
> Name: http-0.0.0.0-8080-213
> State: BLOCKED on org.apache.jackrabbit.core.RepositoryImpl@287ca7 owned by:
> http-0.0.0.0-8080-134
> Total blocked: 1,526 Total waited: 5
>
> Stack trace:
> org.apache.jackrabbit.core.RepositoryImpl.createSession(RepositoryImpl.java)
> org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1090)
> …
>
> Is there something we are doing wrong here?
>
>
> Thanks in advance,
>
> Jerry Lacy
>
>
>
--
-----------------------------------------< tobias.bocanegra@day.com >---
Tobias Bocanegra, Day Management AG, Barfuesserplatz 6, CH - 4001 Basel
T +41 61 226 98 98, F +41 61 226 98 97
-----------------------------------------------< http://www.day.com >---