You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by kevin <ke...@leonardo.com> on 2015/12/31 17:02:43 UTC
Problem with cache expiry and lock
Hi, I ran into a problem with using cache expiry and cache locks.
In my event listener for the expiry, I acquire/release a lock on the key
being expired.
This works fine unless something else waits on the lock after I've acquired
it, then they would be stuck in a deadlock.
The follow code snippet should reproduce this problem.
Should I not be trying to lock the cache object being expired in my
listener?
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setIncludeEventTypes(EventType.EVT_CACHE_OBJECT_EXPIRED);
Ignite ignite = Ignition.start(igniteConfig);
CacheConfiguration<Integer, Integer> cfg = new
CacheConfiguration<>();
cfg.setName("test1");
cfg.setCacheMode(CacheMode.PARTITIONED);
cfg.setRebalanceMode(CacheRebalanceMode.SYNC);
cfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
IgniteCache<Integer, Integer> cache1 = ignite.getOrCreateCache(cfg);
IgniteCache<Integer, Integer> cache2 = cache1.withExpiryPolicy(
new AccessedExpiryPolicy(new Duration(TimeUnit.SECONDS,
1)));
IgnitePredicate<CacheEvent> locallistener = (evt) -> {
try {
Lock lock = cache1.lock(evt.key());
lock.lock();
Thread.sleep(3000); // give some time for other thread to
try to acquire lock
/* -- critical section */
lock.unlock();
} catch (InterruptedException e) {
e.printStackTrace();
}
return true;
};
ignite.events().localListen(locallistener,
EventType.EVT_CACHE_OBJECT_EXPIRED);
cache2.put(1, 1);
Thread.sleep(2000); // wait for cache entry to expire and listener
to run
Lock lock = cache1.lock(1);
lock.lock();
/* -- critical section */
lock.unlock();
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-cache-expiry-and-lock-tp2363.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Problem with cache expiry and lock
Posted by Yakov Zhdanov <yz...@apache.org>.
>From what I see thread cannot acquire lock for segment in cache map. Given
it already holds lock for individual entry and "main" thread has acquired
lock for segment and tries to lock that individual entry (which is proper
order we follow in Ignite). This is canonical deadlock and also one of the
reasons why complex operations are not permitted from listeners called in
sensitive synchronization blocks.
--Yakov
2016-01-04 17:37 GMT+03:00 kevin <ke...@leonardo.com>:
> Thanks Val, your advice has solved the problem for me.
>
> I'm surprised this code worked for you though. Here's a thread dump I took
> if you want to look into it. The listener is stuck on unlocking the key,
> which kinda makes sense to me now that you said the key is already locked.
> I'm using Ignite 1.4
>
> "ttl-cleanup-worker-test-#62%null%":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000642532a78> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap$Segment.putIfObsolete(GridCacheConcurrentMap.java:1044)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMap.java:531)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.entry0(GridCacheAdapter.java:941)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.peekEx(GridCacheAdapter.java:878)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.peekExx(GridDhtCacheAdapter.java:344)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.unlockAll(GridDhtColocatedCache.java:454)
> at
>
> org.apache.ignite.internal.processors.cache.CacheLockImpl.unlock(CacheLockImpl.java:186)
> ----test code
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1181)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:770)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:755)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:295)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:266)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:204)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:3379)
> - locked <0x00000006411b6770> (a
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCacheEntry)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:117)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager$CleanupWorker.body(GridCacheTtlManager.java:136)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> "main":
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.obsolete(GridCacheMapEntry.java:2594)
> - waiting to lock <0x00000006411b6770> (a
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCacheEntry)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap$Segment.putIfObsolete(GridCacheConcurrentMap.java:1074)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMap.java:531)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.entry0(GridCacheAdapter.java:941)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:925)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:363)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryExx(GridDhtCacheAdapter.java:382)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.entryExx(GridDhtColocatedCache.java:167)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.addLocalKey(GridDhtColocatedLockFuture.java:1133)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.mapAsPrimary(GridDhtColocatedLockFuture.java:1092)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.map(GridDhtColocatedLockFuture.java:713)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.mapOnTopology(GridDhtColocatedLockFuture.java:655)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.map(GridDhtColocatedLockFuture.java:607)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.lockAllAsync(GridDhtColocatedCache.java:425)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter.lockAllAsync(GridDistributedCacheAdapter.java:117)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.lockAll(GridCacheAdapter.java:3277)
> at
>
> org.apache.ignite.internal.processors.cache.CacheLockImpl.lock(CacheLockImpl.java:72)
> ----test code
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Problem-with-cache-expiry-and-lock-tp2363p2372.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Problem with cache expiry and lock
Posted by kevin <ke...@leonardo.com>.
Thanks Val, your advice has solved the problem for me.
I'm surprised this code worked for you though. Here's a thread dump I took
if you want to look into it. The listener is stuck on unlocking the key,
which kinda makes sense to me now that you said the key is already locked.
I'm using Ignite 1.4
"ttl-cleanup-worker-test-#62%null%":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000642532a78> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap$Segment.putIfObsolete(GridCacheConcurrentMap.java:1044)
at
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMap.java:531)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.entry0(GridCacheAdapter.java:941)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.peekEx(GridCacheAdapter.java:878)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.peekExx(GridDhtCacheAdapter.java:344)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.unlockAll(GridDhtColocatedCache.java:454)
at
org.apache.ignite.internal.processors.cache.CacheLockImpl.unlock(CacheLockImpl.java:186)
----test code
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1181)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:770)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:755)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:295)
at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:266)
at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:204)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onTtlExpired(GridCacheMapEntry.java:3379)
- locked <0x00000006411b6770> (a
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCacheEntry)
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:117)
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager$CleanupWorker.body(GridCacheTtlManager.java:136)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
"main":
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.obsolete(GridCacheMapEntry.java:2594)
- waiting to lock <0x00000006411b6770> (a
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCacheEntry)
at
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap$Segment.putIfObsolete(GridCacheConcurrentMap.java:1074)
at
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMap.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMap.java:531)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.entry0(GridCacheAdapter.java:941)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:925)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:363)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryExx(GridDhtCacheAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.entryExx(GridDhtColocatedCache.java:167)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.addLocalKey(GridDhtColocatedLockFuture.java:1133)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.mapAsPrimary(GridDhtColocatedLockFuture.java:1092)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.map(GridDhtColocatedLockFuture.java:713)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.mapOnTopology(GridDhtColocatedLockFuture.java:655)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedLockFuture.map(GridDhtColocatedLockFuture.java:607)
at
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.lockAllAsync(GridDhtColocatedCache.java:425)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter.lockAllAsync(GridDistributedCacheAdapter.java:117)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.lockAll(GridCacheAdapter.java:3277)
at
org.apache.ignite.internal.processors.cache.CacheLockImpl.lock(CacheLockImpl.java:72)
----test code
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-cache-expiry-and-lock-tp2363p2372.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Problem with cache expiry and lock
Posted by vkulichenko <va...@gmail.com>.
Hi Kevin,
The event listener is called from the synchronized block on entry, so there
is no reason to use explicit lock there - the key is already locked. And
actually since you're locking the same key, this code should not deadlock
(and it works for me). Probably the issue is caused by some commented out
code inside the critical sections.
But in general, I would not recommend to execute any synchronous cache
operations or acquire locks inside the listener, because it's indeed
deadlock-prone and also can affect performance. If you want to update the
cache after the expiration happened, you should do this asynchronously,
without blocking the thread that called the listener.
Hope this helps.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-cache-expiry-and-lock-tp2363p2365.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.