You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Orange <ja...@blackrock.com> on 2021/05/27 11:20:17 UTC

getOrCreateCache hangs when cacheStore uses spring resources

Calling ignite.getOrCreateCache(cacheConfig) results in the thread getting
blocked. I've noticed this does not happen when the cacheStore does not use
spring resources. Ignite is being started with IgniteSpring.start(config,
ctx).

The are no other server errors.

I've provided the code and the error below.

[2021-05-27 12:11:15.381] - 5368 SEVERE
[tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
org.apache.ignite.internal.util.typedef.G: Blocked system-critical thread
has been detected. This can lead to cluster-wide undefined behaviour
[workerName=partition-exchanger,
threadName=exchange-worker-#46%SERVER-NAME%, blockedFor=1288s]
[2021-05-27 12:11:15.381] - 5368 WARNING
[tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
org.apache.ignite.internal.util.typedef.G: Thread
[name="exchange-worker-#46%SERVER-NAME%", id=87, state=BLOCKED, blockCnt=1,
waitCnt=4]
    Lock [object=java.util.concurrent.ConcurrentHashMap@307677e2,
ownerName=main, ownerId=1]

[2021-05-27 12:11:15.382] - 5368 WARNING
[tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] --- : Possible
failure suppressed accordingly to a configured handler
[hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
heartbeatTs=1622112586746]]]
class org.apache.ignite.IgniteException: GridWorker
[name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
heartbeatTs=1622112586746]
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
	at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
	at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
	at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
	at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970)
	at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057)
	at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086)
	at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995)
	at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

[2021-05-27 12:11:15.382] - 5368 WARNING
[tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
org.apache.ignite.internal.processors.cache.CacheDiagnosticManager: Page
locks dump:

Thread=[name=exchange-worker-#46%SERVER-NAME%, id=87], state=BLOCKED
Locked pages = []
Locked pages log: name=exchange-worker-#46%SERVER-NAME% time=(1622113875382,
2021-05-27 12:11:15.382)


 @Bean
    public CacheConfiguration cacheConfig() {
        CacheConfiguration cacheCfg = new CacheConfiguration("cache-name");
        cacheCfg.setCacheMode(CacheMode.REPLICATED);
        cacheCfg.setReadThrough(true);
        cacheCfg.setCacheStoreFactory(
                new FactoryBuilder.SingletonFactory<>(new
TestCacheStore()));
        return cacheCfg;
    }


public class TestCacheStore extends CacheStoreAdapter<Integer, CacheObject>
        implements Serializable {

    private static final Logger log = getLogger(TestCacheStore .class);

    @SpringResource(resourceName = "serverConfig")
    private transient ServerConfig serverConfig;

    public TestCacheStore () {
        log.info("test");
    }



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: getOrCreateCache hangs when cacheStore uses spring resources

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

This may obviously happen when your Ignite instance is a Spring bean, but
its cache store depends on Spring context.

You need to obtain Spring context but it is still under construction since
Ignite is still being initialized.

Make sure that cache is created only after Spring context is ready and
Ignite initialization has finished.

Regards,
-- 
Ilya Kasnacheev


чт, 27 мая 2021 г. в 14:20, Orange <ja...@blackrock.com>:

> Calling ignite.getOrCreateCache(cacheConfig) results in the thread getting
> blocked. I've noticed this does not happen when the cacheStore does not use
> spring resources. Ignite is being started with IgniteSpring.start(config,
> ctx).
>
> The are no other server errors.
>
> I've provided the code and the error below.
>
> [2021-05-27 12:11:15.381] - 5368 SEVERE
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.util.typedef.G: Blocked system-critical thread
> has been detected. This can lead to cluster-wide undefined behaviour
> [workerName=partition-exchanger,
> threadName=exchange-worker-#46%SERVER-NAME%, blockedFor=1288s]
> [2021-05-27 12:11:15.381] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.util.typedef.G: Thread
> [name="exchange-worker-#46%SERVER-NAME%", id=87, state=BLOCKED, blockCnt=1,
> waitCnt=4]
>     Lock [object=java.util.concurrent.ConcurrentHashMap@307677e2,
> ownerName=main, ownerId=1]
>
> [2021-05-27 12:11:15.382] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] --- :
> Possible
> failure suppressed accordingly to a configured handler
> [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
> heartbeatTs=1622112586746]]]
> class org.apache.ignite.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
> heartbeatTs=1622112586746]
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
>         at
>
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
>         at
>
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>
> [2021-05-27 12:11:15.382] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.processors.cache.CacheDiagnosticManager: Page
> locks dump:
>
> Thread=[name=exchange-worker-#46%SERVER-NAME%, id=87], state=BLOCKED
> Locked pages = []
> Locked pages log: name=exchange-worker-#46%SERVER-NAME%
> time=(1622113875382,
> 2021-05-27 12:11:15.382)
>
>
>  @Bean
>     public CacheConfiguration cacheConfig() {
>         CacheConfiguration cacheCfg = new CacheConfiguration("cache-name");
>         cacheCfg.setCacheMode(CacheMode.REPLICATED);
>         cacheCfg.setReadThrough(true);
>         cacheCfg.setCacheStoreFactory(
>                 new FactoryBuilder.SingletonFactory<>(new
> TestCacheStore()));
>         return cacheCfg;
>     }
>
>
> public class TestCacheStore extends CacheStoreAdapter<Integer, CacheObject>
>         implements Serializable {
>
>     private static final Logger log = getLogger(TestCacheStore .class);
>
>     @SpringResource(resourceName = "serverConfig")
>     private transient ServerConfig serverConfig;
>
>     public TestCacheStore () {
>         log.info("test");
>     }
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: getOrCreateCache hangs when cacheStore uses spring resources

Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hello,

Could you please share log files and a full thread dump from the node?

Thanks,
S.

чт, 27 мая 2021 г. в 14:20, Orange <ja...@blackrock.com>:

> Calling ignite.getOrCreateCache(cacheConfig) results in the thread getting
> blocked. I've noticed this does not happen when the cacheStore does not use
> spring resources. Ignite is being started with IgniteSpring.start(config,
> ctx).
>
> The are no other server errors.
>
> I've provided the code and the error below.
>
> [2021-05-27 12:11:15.381] - 5368 SEVERE
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.util.typedef.G: Blocked system-critical thread
> has been detected. This can lead to cluster-wide undefined behaviour
> [workerName=partition-exchanger,
> threadName=exchange-worker-#46%SERVER-NAME%, blockedFor=1288s]
> [2021-05-27 12:11:15.381] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.util.typedef.G: Thread
> [name="exchange-worker-#46%SERVER-NAME%", id=87, state=BLOCKED, blockCnt=1,
> waitCnt=4]
>     Lock [object=java.util.concurrent.ConcurrentHashMap@307677e2,
> ownerName=main, ownerId=1]
>
> [2021-05-27 12:11:15.382] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] --- :
> Possible
> failure suppressed accordingly to a configured handler
> [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
> heartbeatTs=1622112586746]]]
> class org.apache.ignite.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=SERVER-NAME, finished=false,
> heartbeatTs=1622112586746]
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
>         at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
>         at
>
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
>         at
>
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$1(ServerImpl.java:2970)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8057)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3086)
>         at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
>         at
>
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7995)
>         at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>
> [2021-05-27 12:11:15.382] - 5368 WARNING
> [tcp-disco-msg-worker-[crd]-#2%SERVER-NAME%-#42%SERVER-NAME%] ---
> org.apache.ignite.internal.processors.cache.CacheDiagnosticManager: Page
> locks dump:
>
> Thread=[name=exchange-worker-#46%SERVER-NAME%, id=87], state=BLOCKED
> Locked pages = []
> Locked pages log: name=exchange-worker-#46%SERVER-NAME%
> time=(1622113875382,
> 2021-05-27 12:11:15.382)
>
>
>  @Bean
>     public CacheConfiguration cacheConfig() {
>         CacheConfiguration cacheCfg = new CacheConfiguration("cache-name");
>         cacheCfg.setCacheMode(CacheMode.REPLICATED);
>         cacheCfg.setReadThrough(true);
>         cacheCfg.setCacheStoreFactory(
>                 new FactoryBuilder.SingletonFactory<>(new
> TestCacheStore()));
>         return cacheCfg;
>     }
>
>
> public class TestCacheStore extends CacheStoreAdapter<Integer, CacheObject>
>         implements Serializable {
>
>     private static final Logger log = getLogger(TestCacheStore .class);
>
>     @SpringResource(resourceName = "serverConfig")
>     private transient ServerConfig serverConfig;
>
>     public TestCacheStore () {
>         log.info("test");
>     }
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>