You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Olexandr K <ol...@gmail.com> on 2018/03/06 09:48:16 UTC

How to reconnect Ignite client after server node bounce?

Hi Team,

I tried the following scenario:

1) start local ignite server node
2) start client node and put some key/value pairs
3) bounce server node
4) put some more key/value pairs from client node

at step (4) ignite put(..) operation just hanged

I expected ignite client to take care of auto-reconnection or at least
fail-fast...
How should I handle this?

here is hanged thread's stack:

Daemon Thread [async-worker-2] (Suspended)
    Unsafe.park(boolean, long) line: not available [native method]
    LockSupport.park() line: 304
lock ->
GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get0(boolean)
line: 177
    GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get() line:
140
    GridCacheAdapter$22.op(GridNearTxLocal) line: 2342
    GridCacheAdapter$22.op(GridNearTxLocal) line: 2340

GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).syncOp(GridCacheAdapter<K,SyncOp<>>)
line: 4040
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put0(K, V,
CacheEntryPredicate) line: 2340
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V,
CacheEntryPredicate) line: 2321
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V) line:
2298
    IgniteCacheProxyImpl<K,V>.put(K, V) line: 1005
    GatewayProtectedCacheProxy<K,V>.put(K, V) line: 872
put  ->    PersonRegistrationController.lambda$2(String, String, String,
String) line: 106
    ...

(I'm using ignite 2.3.0)

BR, Oleksandr

Re: How to reconnect Ignite client after server node bounce?

Posted by Olexandr K <ol...@gmail.com>.
Yes, it is reproducible

See client thread dump attached

Server node is just logging metrics, nothing more:

[13:03:10,830][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=d0ae4736, uptime=00:15:00.098]
    ^-- H/N/C [hosts=1, nodes=2, CPUs=2]
    ^-- CPU [cur=0.5%, avg=1.02%, GC=0%]
    ^-- PageMemory [pages=0]
    ^-- Heap [used=67MB, free=72.64%, comm=245MB]
    ^-- Non heap [used=43MB, free=97.12%, comm=44MB]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
    ^-- Outbound messages queue [size=0]


On Tue, Mar 6, 2018 at 12:29 PM, Stanislav Lukyanov <st...@gmail.com>
wrote:

> Hi Olexandr,
>
>
>
> Is it reproducible? Can you share a full thread dump?
>
> Also, are there any related messages in the logs (make sure you’re running
> with `java -DIGNITE_QUEIT=false` or with `ignite.sh -v`)?
>
>
>
> Thanks,
>
> Stan
>
>
>
> *From: *Olexandr K <ol...@gmail.com>
> *Sent: *6 марта 2018 г. 12:48
> *To: *user@ignite.apache.org
> *Subject: *How to reconnect Ignite client after server node bounce?
>
>
>
> Hi Team,
>
> I tried the following scenario:
>
> 1) start local ignite server node
>
> 2) start client node and put some key/value pairs
>
> 3) bounce server node
>
> 4) put some more key/value pairs from client node
>
>
>
> at step (4) ignite put(..) operation just hanged
>
> I expected ignite client to take care of auto-reconnection or at least
> fail-fast...
>
> How should I handle this?
>
>
>
> here is hanged thread's stack:
>
> Daemon Thread [async-worker-2] (Suspended)
>     Unsafe.park(boolean, long) line: not available [native method]
>     LockSupport.park() line: 304
> lock ->    GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get0(boolean)
> line: 177
>     GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get() line:
> 140
>     GridCacheAdapter$22.op(GridNearTxLocal) line: 2342
>     GridCacheAdapter$22.op(GridNearTxLocal) line: 2340
>     GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).syncOp(GridCacheAdapter<K,SyncOp<>>)
> line: 4040
>     GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put0(K, V,
> CacheEntryPredicate) line: 2340
>     GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V,
> CacheEntryPredicate) line: 2321
>     GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V) line:
> 2298
>     IgniteCacheProxyImpl<K,V>.put(K, V) line: 1005
>     GatewayProtectedCacheProxy<K,V>.put(K, V) line: 872
> put  ->    PersonRegistrationController.lambda$2(String, String, String,
> String) line: 106
>     ...
>
> (I'm using ignite 2.3.0)
>
>
>
> BR, Oleksandr
>
>
>

RE: How to reconnect Ignite client after server node bounce?

Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi Olexandr,

Is it reproducible? Can you share a full thread dump? 
Also, are there any related messages in the logs (make sure you’re running with `java -DIGNITE_QUEIT=false` or with `ignite.sh -v`)? 

Thanks,
Stan

From: Olexandr K
Sent: 6 марта 2018 г. 12:48
To: user@ignite.apache.org
Subject: How to reconnect Ignite client after server node bounce?

Hi Team,
I tried the following scenario:
1) start local ignite server node
2) start client node and put some key/value pairs
3) bounce server node
4) put some more key/value pairs from client node

at step (4) ignite put(..) operation just hanged
I expected ignite client to take care of auto-reconnection or at least fail-fast...
How should I handle this?

here is hanged thread's stack:

Daemon Thread [async-worker-2] (Suspended)    
    Unsafe.park(boolean, long) line: not available [native method]    
    LockSupport.park() line: 304    
lock ->    GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get0(boolean) line: 177    
    GridFutureAdapter$ChainFuture<R,T>(GridFutureAdapter<R>).get() line: 140    
    GridCacheAdapter$22.op(GridNearTxLocal) line: 2342    
    GridCacheAdapter$22.op(GridNearTxLocal) line: 2340    
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).syncOp(GridCacheAdapter<K,SyncOp<>>) line: 4040    
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put0(K, V, CacheEntryPredicate) line: 2340    
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V, CacheEntryPredicate) line: 2321    
    GridDhtColocatedCache<K,V>(GridCacheAdapter<K,V>).put(K, V) line: 2298    
    IgniteCacheProxyImpl<K,V>.put(K, V) line: 1005    
    GatewayProtectedCacheProxy<K,V>.put(K, V) line: 872    
put  ->    PersonRegistrationController.lambda$2(String, String, String, String) line: 106    
    ...    
(I'm using ignite 2.3.0)

BR, Oleksandr