Ignite Stopping Abruptly

[Gurmehar Kalra <gu...@hcl.com>]

Hi,

Using Apache Ignite 2.10, with config as

    cache.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
    cache.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
    cache.setCacheMode(CacheMode.REPLICATED);
    cache.setGroupName("EngineGroup");
    cache.setOnheapCacheEnabled(true);
    cache.setEvictionPolicyFactory(()-> new LruEvictionPolicy(10000));

There are 2 cache which have TTL enabled
kafkaCacheError.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.TWENTY_MINUTES));
kafkaCacheError.setEagerTtl(true);

ignite give intermittent error
'Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=ttl-cleanup-worker, igniteInstanceName=journeyengine, finished=false, heartbeatTs=1625507061348]]]

after sometime of this error ignite stops .

Please suggest

Regards,
Gurmehar Singh

::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
________________________________

Re: Ignite Stopping Abruptly

[Alexey Kukushkin <ku...@gmail.com>]

You enabled eager TTL, which means you have a background thread that cleans
expired entries. The message you copied from the log says the TTL thread
was blocked for more than IgniteConfiguration/sysWorkerBlockedTimeout,
which by default equals  IgniteConfiguration/failureDetectionTimeout, which
by default is 10 seconds.

Most likely that happened due to high resource (CPU or memory) usage so
that the TTL thread was not given a chance to execute for 10 seconds. If
that is the case, you need to either scale up the cluster sieze or optimize
your app or just increase the sysWorkerBlockedTimeout if you are OK with
more than 10 seconds expiration latency.



вт, 6 июл. 2021 г. в 08:56, Gurmehar Kalra <gu...@hcl.com>:

> Hi,
>
>
>
> Using Apache Ignite 2.10, with config as
>
>
>
>     *cache**.setWriteSynchronizationMode(CacheWriteSynchronizationMode.*
> *FULL_ASYNC**);*
>
>     *cache**.setAtomicityMode(CacheAtomicityMode.**TRANSACTIONAL**);*
>
>     *cache**.setCacheMode(CacheMode.**REPLICATED**);*
>
>     *cache**.setGroupName(**"EngineGroup"**);*
>
>     *cache**.setOnheapCacheEnabled(**true**);*
>
>     *cache**.setEvictionPolicyFactory(()-> **new**
> LruEvictionPolicy(10000))**;*
>
>
>
> There are 2 cache which have TTL enabled
>
> *kafkaCacheError*
> *.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(Duration.*
> *TWENTY_MINUTES**));*
>
> *kafkaCacheError**.setEagerTtl(**true**);*
>
>
>
> ignite give intermittent error
> *‘Possible failure suppressed accordingly to a configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=ttl-cleanup-worker,
> igniteInstanceName=journeyengine, finished=false,
> heartbeatTs=1625507061348]]]*
>
>
>
> after sometime of this error ignite stops .
>
>
>
> Please suggest
>
>
>
> Regards,
>
> Gurmehar Singh
>
>
> ::DISCLAIMER::
> ------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only. E-mail transmission is not
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or may contain
> viruses in transmission. The e mail and its contents (with or without
> referred errors) shall therefore not attach any liability on the originator
> or HCL or its affiliates. Views or opinions, if any, presented in this
> email are solely those of the author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification, distribution and / or
> publication of this message without the prior written consent of authorized
> representative of HCL is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately. Before
> opening any email and/or attachments, please check them for viruses and
> other defects.
> ------------------------------
>


-- 
Best regards,
Alexey