[jira] Updated: (JCR-2554) Deadlock inside XASession on Weblogic

["Robert Sauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/JCR-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Sauer updated JCR-2554:
------------------------------

    Attachment: jr-core-xid-aware-ism-locking1.patch

I tried to be conservative and not replace DefaultISMLocking. Instead I just installed a special ISMLocking instance for AbstractVersionManager which uses the XID aware implementation if an XID is present or falls back to DefaultISMLocking otherwise.

This patch was created against the 1.6.1 tag. Test for both jackrabbit-core and jackrabbit-jca are passing.

The initial implementation was using java.util.concurrent APIs and my ad hoc tests indicate this had a somewhat higher througput than the oswego-concurrent based implementation contained in this patch (200 TX/s vs 165 TX/s).

With this patch applied my test driver could repeatedly perform about 45.000 TXs from 45 concurrent threads without deadlocking connections.

> Deadlock inside XASession on Weblogic
> -------------------------------------
>
>                 Key: JCR-2554
>                 URL: https://issues.apache.org/jira/browse/JCR-2554
>             Project: Jackrabbit Content Repository
>          Issue Type: Bug
>          Components: jackrabbit-core
>    Affects Versions: 1.6.1, 2.0.0
>         Environment: Weblogic 9.2
>            Reporter: Robert Sauer
>            Assignee: Claus Köll
>            Priority: Critical
>         Attachments: jr-core-xid-aware-ism-locking1.patch
>
>
> In one of our client deployments on WebLogic 9.2 we observed JackRabbit sessions going stale in a load test. This was observed against release 1.6.1 (to which we migrated due to concurrency related issues JCR-2081 and JCR-2237). Same effect with 2.0.0.
>  
> I could finally reproduce this issue locally. And it seems to boil down to WLS invoking the sequence of <prepare> ... <release> ... <commit> on one XA session from multiple threads, as it seems breaking assumptions of the thread-bound java.util.concurrent-RWLock based DefaultISMLocking class.
> Effectively the setActiveXid(..) method on DefaultISMLocking$RWLock fails as the old active XID was not yet cleared. With the result of more and more sessions deadlocking in below's invocation stack.
> {code}
> "[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=1 tid=0x33fc3ec0 nid=0x2324 in Object.wait() [0x2156a000..0x2156beb0] at java.lang.Object.wait(Native Method) - waiting on <0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock) at java.lang.Object.wait(Object.java:474) at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(Unknown Source) - locked <0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock) at org.apache.jackrabbit.core.state.DefaultISMLocking$1.<init>(DefaultISMLocking.java:64) at org.apache.jackrabbit.core.state.DefaultISMLocking.acquireWriteLock(DefaultISMLocking.java:61) at org.apache.jackrabbit.core.version.AbstractVersionManager.acquireWriteLock(AbstractVersionManager.java:146) at org.apache.jackrabbit.core.version.XAVersionManager$1.prepare(XAVersionManager.java:562) at org.apache.jackrabbit.core.TransactionContext.prepare(TransactionContext.java:154) - locked <0x6dc2ad88> (a org.apache.jackrabbit.core.TransactionContext) at org.apache.jackrabbit.core.XASessionImpl.prepare(XASessionImpl.java:331) at org.apache.jackrabbit.jca.TransactionBoundXAResource.prepare(TransactionBoundXAResource.java:68) at weblogic.connector.security.layer.AdapterLayer.prepare(AdapterLayer.java:397) at weblogic.connector.transaction.outbound.XAWrapper.prepare(XAWrapper.java:297) at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:1276) at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:499) at weblogic.transaction.internal.ServerSCInfo$1.execute(ServerSCInfo.java:335) at weblogic.kernel.Kernel.executeIfIdle(Kernel.java:243) at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:326) at weblogic.transaction.internal.ServerTransactionImpl.localPrepare(ServerTransactionImpl.java:2516) at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2211) at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:266) at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227) at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:283) at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.