You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by aasoj j <aa...@gmail.com> on 2009/07/28 11:30:15 UTC
Deadlock in cluster deployment
Hi,
We have a 3 host Jackrabbit cluster deployment (version 1.5.4). In our case
new data nodes are added to the repository frequently (500 new nodes in
different parts of the tree every 15 minutes), and a huge number of reads
are serviced (50 per second).
After working for a few hours we observe that most of the threads end up in
a wait state:
"http-0.0.0.0-4080-37" Id=3098 in WAITING on
lock=EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@42ae9b4d
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(Unknown
Source)
at
org.apache.jackrabbit.core.state.DefaultISMLocking$ReadLockImpl.<init>(DefaultISMLocking.java:84)
at
org.apache.jackrabbit.core.state.DefaultISMLocking$ReadLockImpl.<init>(DefaultISMLocking.java:78)
at
org.apache.jackrabbit.core.state.DefaultISMLocking.acquireReadLock(DefaultISMLocking.java:44)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(SharedItemStateManager.java:1409)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.hasItemState(SharedItemStateManager.java:286)
at
org.apache.jackrabbit.core.state.XAItemStateManager.hasItemState(XAItemStateManager.java:295)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:181)
at
org.apache.jackrabbit.core.HierarchyManagerImpl.getItemState(HierarchyManagerImpl.java:150)
at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:113)
at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:325)
at
org.apache.jackrabbit.core.CachingHierarchyManager.resolvePath(CachingHierarchyManager.java:135)
at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:363)
at org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:435)
at org.apache.jackrabbit.core.SessionImpl.getItem(SessionImpl.java:817)
This is usually observed after new nodes are added to the repository in one
of the host.
We also observed the following NPE once.
ERROR org.apache.jackrabbit.core.cluster.ClusterNode:292 Unexpected error
while syncing of journal: null
java.lang.NullPointerException
at
org.apache.jackrabbit.core.state.XAItemStateManager.stateModified(XAItemStateManager.java:580)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(StateChangeDispatcher.java:111)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.stateModified(SharedItemStateManager.java:400)
at
org.apache.jackrabbit.core.state.ItemState.notifyStateUpdated(ItemState.java:244)
at
org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:285)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.doExternalUpdate(SharedItemStateManager.java:1186)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.externalUpdate(SharedItemStateManager.java:1109)
at
org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.externalUpdate(RepositoryImpl.java:2098)
at
org.apache.jackrabbit.core.cluster.ClusterNode.process(ClusterNode.java:852)
at
org.apache.jackrabbit.core.cluster.ChangeLogRecord.process(ChangeLogRecord.java:384)
at
org.apache.jackrabbit.core.cluster.ClusterNode.consume(ClusterNode.java:798)
at
org.apache.jackrabbit.core.journal.AbstractJournal.doSync(AbstractJournal.java:213)
at
org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:188)
at
org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:315)
at
org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:286)
at java.lang.Thread.run(Thread.java:619)
It seems that these issues are similar to issues below.
1.
http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
.
2. https://issues.apache.org/jira/browse/JCR-2171
Please advise.
Thanks in advance
Re: Deadlock in cluster deployment
Posted by aasoj j <aa...@gmail.com>.
Additionally the cluster synchronization also stopped after this exception.
I observed the following in the thread dump
"ClusterNode-02" Id=20 in WAITING on
lock=EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@7f6b5599
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(Unknown
Source)
at
org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:182)
at
org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:315)
at
org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:286)
at java.lang.Thread.run(Thread.java:619)
On Mon, Aug 3, 2009 at 11:29 AM, aasoj j <aa...@gmail.com> wrote:
> Thanks Marcel,
>
> I downloaded 1.5.7 src and applied the the patch (
> https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch).
> This solves the problem. However now I can see a new NPE. Please see the
> stack trace below.
>
> java.lang.NullPointerException
> at
> org.apache.jackrabbit.core.ItemManager.retrieveItem(ItemManager.java:743)
> at
> org.apache.jackrabbit.core.ItemManager.stateCreated(ItemManager.java:927)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:812)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
> at
> org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:235)
> at
> org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:310)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:751)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1100)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager$WriteOperation.save(AbstractVersionManager.java:150)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager.createVersionHistory(AbstractVersionManager.java:317)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl$1.run(VersionManagerImpl.java:220)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl$DynamicESCFactory.doSourced(VersionManagerImpl.java:567)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl.createVersionHistory(VersionManagerImpl.java:217)
> at
> org.apache.jackrabbit.core.version.XAVersionManager.createVersionHistory(XAVersionManager.java:156)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager.getVersionHistory(AbstractVersionManager.java:234)
> at
> org.apache.jackrabbit.core.ItemImpl.initVersionHistories(ItemImpl.java:727)
> at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1080)
> at
> org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:858)
>
>
> regards
> aasoj
>
>
> On Thu, Jul 30, 2009 at 3:44 PM, Marcel Reutegger <
> marcel.reutegger@gmx.net> wrote:
>
>> Hi,
>>
>> On Tue, Jul 28, 2009 at 11:30, aasoj j<aa...@gmail.com> wrote:
>> > It seems that these issues are similar to issues below.
>> >
>> > 1.
>> >
>> http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
>>
>> this has been fixed in 1.5.7. the release is not yet listed on the
>> download page but you can still download it:
>>
>> http://www.apache.org/dyn/closer.cgi/jackrabbit/1.5.7/source/jackrabbit-1.5.7-src.jar
>>
>> > 2. https://issues.apache.org/jira/browse/JCR-2171
>>
>> that's been fixed as well but is not yet available in a release. as
>> reported by Sébastien Launay patching the 1.5.7 release works. See
>> https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch
>> for the required change.
>>
>> If that's not viable for you then you either have to wait for the 1.6
>> release or request a 1.5.8 release that will include the fix for
>> JCR-2171.
>>
>> regards
>> marcel
>>
>
>
Re: Deadlock in cluster deployment
Posted by Marcel Reutegger <ma...@gmx.net>.
Hi,
I guess this is similar to JCR-2168 but for ItemManager. The following
patch might solve the problem:
Index: src/main/java/org/apache/jackrabbit/core/ItemManager.java
===================================================================
--- src/main/java/org/apache/jackrabbit/core/ItemManager.java (revision 800265)
+++ src/main/java/org/apache/jackrabbit/core/ItemManager.java (working copy)
@@ -119,12 +119,12 @@
this.rootNodeDef = rootNodeDef;
this.rootNodeId = rootNodeId;
+ // setup shareable nodes cache
+ shareableNodesCache = new ShareableNodesCache();
+
// setup item cache with weak references to items
itemCache = new ReferenceMap(ReferenceMap.HARD, ReferenceMap.WEAK);
itemStateProvider.addListener(this);
-
- // setup shareable nodes cache
- shareableNodesCache = new ShareableNodesCache();
}
/**
I've created a jira issue for this:
https://issues.apache.org/jira/browse/JCR-2237
regards
marcel
On Mon, Aug 3, 2009 at 07:59, aasoj j<aa...@gmail.com> wrote:
> Thanks Marcel,
>
> I downloaded 1.5.7 src and applied the the patch (
> https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch).
> This solves the problem. However now I can see a new NPE. Please see the
> stack trace below.
>
> java.lang.NullPointerException
> at
> org.apache.jackrabbit.core.ItemManager.retrieveItem(ItemManager.java:743)
> at
> org.apache.jackrabbit.core.ItemManager.stateCreated(ItemManager.java:927)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:812)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
> at
> org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
> at
> org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:235)
> at
> org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:310)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:751)
> at
> org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1100)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351)
> at
> org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager$WriteOperation.save(AbstractVersionManager.java:150)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager.createVersionHistory(AbstractVersionManager.java:317)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl$1.run(VersionManagerImpl.java:220)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl$DynamicESCFactory.doSourced(VersionManagerImpl.java:567)
> at
> org.apache.jackrabbit.core.version.VersionManagerImpl.createVersionHistory(VersionManagerImpl.java:217)
> at
> org.apache.jackrabbit.core.version.XAVersionManager.createVersionHistory(XAVersionManager.java:156)
> at
> org.apache.jackrabbit.core.version.AbstractVersionManager.getVersionHistory(AbstractVersionManager.java:234)
> at
> org.apache.jackrabbit.core.ItemImpl.initVersionHistories(ItemImpl.java:727)
> at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1080)
> at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:858)
>
>
> regards
> aasoj
>
> On Thu, Jul 30, 2009 at 3:44 PM, Marcel Reutegger
> <ma...@gmx.net>wrote:
>
>> Hi,
>>
>> On Tue, Jul 28, 2009 at 11:30, aasoj j<aa...@gmail.com> wrote:
>> > It seems that these issues are similar to issues below.
>> >
>> > 1.
>> >
>> http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
>>
>> this has been fixed in 1.5.7. the release is not yet listed on the
>> download page but you can still download it:
>>
>> http://www.apache.org/dyn/closer.cgi/jackrabbit/1.5.7/source/jackrabbit-1.5.7-src.jar
>>
>> > 2. https://issues.apache.org/jira/browse/JCR-2171
>>
>> that's been fixed as well but is not yet available in a release. as
>> reported by Sébastien Launay patching the 1.5.7 release works. See
>> https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch
>> for the required change.
>>
>> If that's not viable for you then you either have to wait for the 1.6
>> release or request a 1.5.8 release that will include the fix for
>> JCR-2171.
>>
>> regards
>> marcel
>>
>
Re: Deadlock in cluster deployment
Posted by aasoj j <aa...@gmail.com>.
Thanks Marcel,
I downloaded 1.5.7 src and applied the the patch (
https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch).
This solves the problem. However now I can see a new NPE. Please see the
stack trace below.
java.lang.NullPointerException
at
org.apache.jackrabbit.core.ItemManager.retrieveItem(ItemManager.java:743)
at
org.apache.jackrabbit.core.ItemManager.stateCreated(ItemManager.java:927)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
at
org.apache.jackrabbit.core.state.SessionItemStateManager.stateCreated(SessionItemStateManager.java:812)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.stateCreated(LocalItemStateManager.java:428)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateCreated(StateChangeDispatcher.java:96)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.stateCreated(SharedItemStateManager.java:390)
at
org.apache.jackrabbit.core.state.ItemState.notifyStateCreated(ItemState.java:235)
at
org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:310)
at
org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:751)
at
org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1100)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:351)
at
org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
at
org.apache.jackrabbit.core.version.AbstractVersionManager$WriteOperation.save(AbstractVersionManager.java:150)
at
org.apache.jackrabbit.core.version.AbstractVersionManager.createVersionHistory(AbstractVersionManager.java:317)
at
org.apache.jackrabbit.core.version.VersionManagerImpl$1.run(VersionManagerImpl.java:220)
at
org.apache.jackrabbit.core.version.VersionManagerImpl$DynamicESCFactory.doSourced(VersionManagerImpl.java:567)
at
org.apache.jackrabbit.core.version.VersionManagerImpl.createVersionHistory(VersionManagerImpl.java:217)
at
org.apache.jackrabbit.core.version.XAVersionManager.createVersionHistory(XAVersionManager.java:156)
at
org.apache.jackrabbit.core.version.AbstractVersionManager.getVersionHistory(AbstractVersionManager.java:234)
at
org.apache.jackrabbit.core.ItemImpl.initVersionHistories(ItemImpl.java:727)
at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1080)
at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:858)
regards
aasoj
On Thu, Jul 30, 2009 at 3:44 PM, Marcel Reutegger
<ma...@gmx.net>wrote:
> Hi,
>
> On Tue, Jul 28, 2009 at 11:30, aasoj j<aa...@gmail.com> wrote:
> > It seems that these issues are similar to issues below.
> >
> > 1.
> >
> http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
>
> this has been fixed in 1.5.7. the release is not yet listed on the
> download page but you can still download it:
>
> http://www.apache.org/dyn/closer.cgi/jackrabbit/1.5.7/source/jackrabbit-1.5.7-src.jar
>
> > 2. https://issues.apache.org/jira/browse/JCR-2171
>
> that's been fixed as well but is not yet available in a release. as
> reported by Sébastien Launay patching the 1.5.7 release works. See
> https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch
> for the required change.
>
> If that's not viable for you then you either have to wait for the 1.6
> release or request a 1.5.8 release that will include the fix for
> JCR-2171.
>
> regards
> marcel
>
Re: Deadlock in cluster deployment
Posted by Marcel Reutegger <ma...@gmx.net>.
Hi,
On Tue, Jul 28, 2009 at 11:30, aasoj j<aa...@gmail.com> wrote:
> It seems that these issues are similar to issues below.
>
> 1.
> http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
this has been fixed in 1.5.7. the release is not yet listed on the
download page but you can still download it:
http://www.apache.org/dyn/closer.cgi/jackrabbit/1.5.7/source/jackrabbit-1.5.7-src.jar
> 2. https://issues.apache.org/jira/browse/JCR-2171
that's been fixed as well but is not yet available in a release. as
reported by Sébastien Launay patching the 1.5.7 release works. See
https://issues.apache.org/jira/secure/attachment/12412869/JCR-2171.patch
for the required change.
If that's not viable for you then you either have to wait for the 1.6
release or request a 1.5.8 release that will include the fix for
JCR-2171.
regards
marcel