You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by DanieleBosetti <D....@cmcmarkets.com> on 2018/10/10 11:35:39 UTC
Call to getCache never returns on CLIENT_NODE_RECONNECTED event
Hi,
I am trying to implement a mechanism so that clients can recover from a full
cluster shutdown.
Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then
tries issuing another query.
The reconnection handling code is executed by a "disco-event-worker-#..."
thread; and when executing IgniteKernal.getOrCreateCache (from the discovery
thread); the call never returns. (stacktrace below).
If the very same code is executed asynchronously (not in this
"discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes
normally.
I suspect the cause of this problem is that there some sort of deadlock,
caused by calling getOrCreateCache from this thread before the reconnection
event (that we are processing) is forwarded to all other listeners.
Would this make sense/is that expected?
Thanks!
*Register to the reconnection event*
KeepAliveQueryListener keepAliveListener = new
KeepAliveQueryListener(ignite, queryBuilder, listener);
ignite.events().localListen(keepAliveListener,
EventType.EVT_CLIENT_NODE_RECONNECTED);
*event handling:*
class KeepAliveQueryListener { ...
@Override
public boolean apply(DiscoveryEvent de) {
exec(() -> {
logger.info("on event: type={} ", de.name());
if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) {
logger.warn("restarting CQ on client reconnection..");
startQuery();
logger.warn("restarted CQ");
}
});
return true;
}
}
private static final Executor es = Executors.newFixedThreadPool(4);
private void exec(Runnable r) {
// asynch vs sync
r.run();
//es.execute(r);
}
class QueryBuilder { ...
public IgniteCache<BinaryObject, BinaryObject> getCache() {
logger.info("getCache name={} called..", table);
IgniteCache<BinaryObject, BinaryObject> res =
ignite.getOrCreateCache(table).withKeepBinary();
logger.info("getCache name={} done", table);
return res;
}
}
*stacktrace*
"disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0
tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107)
at cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85)
at
cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43)
at
cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67)
at
cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown
Source)
at
cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56)
at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62)
at
cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Call to getCache never returns on CLIENT_NODE_RECONNECTED event
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Hi,
Yes, it is a bad idea to do such cache operations in discovery thread and
this may lead to the deadlock. You can start a different thread if you want
to do these operations.
Evgenii
ср, 10 окт. 2018 г. в 14:35, DanieleBosetti <D....@cmcmarkets.com>:
> Hi,
>
> I am trying to implement a mechanism so that clients can recover from a
> full
> cluster shutdown.
> Basically on CLIENT_RECONNECTION, my code closes a QueryCursor and then
> tries issuing another query.
>
> The reconnection handling code is executed by a "disco-event-worker-#..."
> thread; and when executing IgniteKernal.getOrCreateCache (from the
> discovery
> thread); the call never returns. (stacktrace below).
>
> If the very same code is executed asynchronously (not in this
> "discovery-worker" thread), then the IgniteKernal.getOrCreateCache executes
> normally.
>
> I suspect the cause of this problem is that there some sort of deadlock,
> caused by calling getOrCreateCache from this thread before the reconnection
> event (that we are processing) is forwarded to all other listeners.
>
> Would this make sense/is that expected?
>
>
> Thanks!
>
>
> *Register to the reconnection event*
>
> KeepAliveQueryListener keepAliveListener = new
> KeepAliveQueryListener(ignite, queryBuilder, listener);
> ignite.events().localListen(keepAliveListener,
> EventType.EVT_CLIENT_NODE_RECONNECTED);
>
> *event handling:*
>
> class KeepAliveQueryListener { ...
> @Override
> public boolean apply(DiscoveryEvent de) {
> exec(() -> {
> logger.info("on event: type={} ", de.name());
> if (de.type() == EventType.EVT_CLIENT_NODE_RECONNECTED) {
> logger.warn("restarting CQ on client reconnection..");
> startQuery();
> logger.warn("restarted CQ");
> }
> });
> return true;
> }
> }
> private static final Executor es = Executors.newFixedThreadPool(4);
> private void exec(Runnable r) {
> // asynch vs sync
> r.run();
> //es.execute(r);
> }
>
> class QueryBuilder { ...
> public IgniteCache<BinaryObject, BinaryObject> getCache() {
> logger.info("getCache name={} called..", table);
> IgniteCache<BinaryObject, BinaryObject> res =
> ignite.getOrCreateCache(table).withKeepBinary();
> logger.info("getCache name={} done", table);
> return res;
> }
> }
>
> *stacktrace*
> "disco-event-worker-#122%listener-client%" #167 prio=5 os_prio=0
> tid=0x000000002019f000 nid=0x7934 waiting on condition [0x0000000040fbe000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
>
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3107)
> at
> cachelistener.impl.ignite.QueryBuilder.getCache(QueryBuilder.java:85)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener.startQuery(KeepAliveQueryListener.java:43)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener.lambda$0(KeepAliveQueryListener.java:67)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener$$Lambda$28/1019213101.run(Unknown
> Source)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener.exec(KeepAliveQueryListener.java:56)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:62)
> at
>
> cachelistener.impl.ignite.KeepAliveQueryListener.apply(KeepAliveQueryListener.java:1)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1477)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:873)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:858)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
> at
>
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
> at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2703)
> at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2920)
> at
>
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2732)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>