You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by styriver <Sc...@mgic.com> on 2017/01/16 22:44:29 UTC
Locking behavior
I have two questions about locking.
1) Is it expected behavior that if an application fails to call an unlock it
locks the entire cache and not just that object?
We have a web application that had a bug where unlock was not being called.
But the behavior that we experienced is that this affected all users. All
users were "hung" until we restarted both server nodes to free the lock. The
cache in question uses the userid as a key. So it appears we lock the entire
cache not just a particular map entry. When we ask for the lock we are
asking based on the map key?
Lock lock = getCache().lock(userId);
try {
if (getCache().containsKey(userId)) {
lockedObj = new LockedObj(lock, getCache().get(userId));
//acquire lock
lock.lock();
}
2) Is there a way to "force" an unlock in the event. The only way I could
figure out to resolve the issues was to take down both server nodes that
hold the cache and restart them.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Locking-behavior-tp10087.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Locking behavior
Posted by Yakov Zhdanov <yz...@apache.org>.
This is the trace of the waiting thread. This means there is some thread in
the cluster that holds the lock. Is it possible to determine which one and
on which jvm and why this lock has not been released.
> We did not use transactions because we did not want to have to handle the
condition of what to do if the commit fails. We thought with locking if two
threads were updating at the same time the other would just wait until it
became free.
I am afraid I don't 100% understand this. Transactions were designed and
implemented exactly for that goal - if commit fails then nothing changes
and data remains consistent. And what do you do if cache.put() fails? Btw,
put() in transactional cache starts implicit transaction :) so you have
transactions anyways.
--Yakov
Re: Locking behavior
Posted by styriver <Sc...@mgic.com>.
We have a fairly complex web app in terms of business logic. We are replacing
apache JCs with this product. We did not use transactions because we did not
want to have to handle the condition of what to do if the commit fails. We
thought with locking if two threads were updating at the same time the other
would just wait until it became free.
We are acquiring the lock and then executing numerous updates to the
"locked" object via numerous method calls. When complete we are saving it
back to the cache and unlocking.
Here is a stack trace from our Tomcat Stuck Thread Detection
Jan 10, 2017 9:24:57 AM org.apache.catalina.valves.StuckThreadDetectionValve
notifyStuckThreadDetected
WARNING: Thread "xxxxx" (id=221) has been active for 61,616 milliseconds
(since 1/10/17 9:23 AM) to serve the same request for
https://xx.xxxx.xxx.net/busacq-xxxx/closeLoan.htm and may be stuck
(configured threshold for this StuckThreadDetectionValve is 60 seconds).
There is/are 1 thread(s) in total that are monitored by this Valve and may
be stuck.
java.lang.Throwable
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:158)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:118)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.lockAll(GridCacheAdapter.java:3310)
at
org.apache.ignite.internal.processors.cache.CacheLockImpl.lock(CacheLockImpl.java:74)
at
com.xxx.xxxx.service.cache.LoanContainerCacheServiceImpl.fetchWithLock(LoanContainerCacheServiceImpl.java:163)
at
com.xxx.xxxx.service.AtlasObjServiceImpl.fetchModifyableAtlasLoanContainer(AtlasObjServiceImpl.java:169)
at
com.xxx.xxxx.session.AtlasSessionFilter.doFilter(AtlasSessionFilter.java:93)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
com.xxx.security.cas.ExtranetAuthorizationFilter.doFilter(ExtranetAuthorizationFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:75)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:201)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.jasig.cas.client.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:107)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
com.xxx.security.cas.ExtranetAuthenticationFilterConfigurator.doFilter(ExtranetAuthenticationFilterConfigurator.java:98)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:178)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at
org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:221)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:333)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at
org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2459)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Locking-behavior-tp10087p10096.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Locking behavior
Posted by Yakov Zhdanov <yz...@apache.org>.
1. Lock is acquired on per entry (per-key) basis. Can you please provide
threaddumps and logs from all of the nodes. Note, that locks are bound to
the thread they are acquired in.
2. It seems the answer is NO. One option though - you can stop the node
holding the lock, i.e. node where cache.lock(key).lock(); succeeded. In
this case lock will be released on server node and next candidate should
succeed.
Can you please explain why you use cache locks. In most cases cache locks
in code can be replaced with cache transactions and it seems to me you try
to replace tx locking with explicit locks which looks like overhead to me.
Use try-with-resources notation to work with txs. This should prevent you
from having unreleased locks.
--Yakov
2017-01-17 1:44 GMT+03:00 styriver <Sc...@mgic.com>:
> I have two questions about locking.
>
> 1) Is it expected behavior that if an application fails to call an unlock
> it
> locks the entire cache and not just that object?
>
> We have a web application that had a bug where unlock was not being called.
> But the behavior that we experienced is that this affected all users. All
> users were "hung" until we restarted both server nodes to free the lock.
> The
> cache in question uses the userid as a key. So it appears we lock the
> entire
> cache not just a particular map entry. When we ask for the lock we are
> asking based on the map key?
>
> Lock lock = getCache().lock(userId);
>
> try {
> if (getCache().containsKey(userId)) {
> lockedObj = new LockedObj(lock, getCache().get(userId));
> //acquire lock
> lock.lock();
> }
>
> 2) Is there a way to "force" an unlock in the event. The only way I could
> figure out to resolve the issues was to take down both server nodes that
> hold the cache and restart them.
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Locking-behavior-tp10087.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>