You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alper Tekinalp <al...@evam.com> on 2016/10/25 07:32:11 UTC
Threads got stuck
Hi all.
We have 3 servers and cache configuration like:
<bean class="org.apache.ignite.configuration.CacheConfiguration"
name="DEFAULT">
<property name="atomicityMode" value="ATOMIC" />
<property name="cacheMode" value="PARTITIONED" />
<property name="memoryMode" value="OFFHEAP_TIERED" />
<property name="backups" value="1" />
<property name="affinity">
<bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">
<constructor-arg index="0" type="int"
value="#{evamProperties['topology.cache.partition.size']}"/>
</bean>
</property>
<!-- max memory is set to 0 means unlimited -->
<property name="offHeapMaxMemory" value="0" />
<property name="writeThrough" value="false" />
<property name="writeBehindEnabled" value="false" />
<property name="eagerTtl" value="false"/>
</bean>
For our worker threads we check heartbeat and if a thread did not sent
heart beat for 10 minutes we consider it as stucked and interrrupt and
recreate it.
As I can see all our worker threads are stucked in cache.put() state and
interrupted and recreated regularly.
What can be the reason we are stucked at put? Following is stacktrace for
interruption error.
javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Failed to wait for
asynchronous operation permit (thread got interrupted).
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1502)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2021)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
at
com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(ConfigManagerHelperEE.java:90)
at
com.intellica.project.helper.ee.StateMachineConfigManagerEEImpl.store(StateMachineConfigManagerEEImpl.java:53)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.storeConfig(AbstractEventProcessor.java:175)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:130)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:80)
at
com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine.java:199)
at
com.evelopers.unimod.runtime.StrictHandler.handle(StrictHandler.java:46)
at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenarioLogic(AbstractScenarioWorker.java:172)
at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenario(AbstractScenarioWorker.java:130)
at
com.intellica.evam.engine.server.worker.AsyncWorker.processEvent(AsyncWorker.java:156)
at
com.intellica.evam.engine.server.worker.AsyncWorker.run(AsyncWorker.java:88)
Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
wait for asynchronous operation permit (thread got interrupted).
at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:747)
at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:745)
... 14 more
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
... 11 more
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
After restarting servers, that issue did not occur again. I guess for some
reason the cluster did not start normally.
Thanks responding so quick.
Regards.
On Tue, Oct 25, 2016 at 11:31 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> Alper, thanks for clarification, this will definitely help after we get the
> info I requested. This is the only way to go with the investigation.
>
> --Yakov
>
> 2016-10-25 11:20 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>
> > Hi Yakov.
> >
> > I should also mention that we load cache data from one server and wait
> the
> > data to be replicated to others. Can that cause such a situation, too?
> >
> > On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org>
> > wrote:
> >
> >> Alper,
> >>
> >> There can be multiple reasons.
> >>
> >> Can you please reproduce the issue one more time, collect and share the
> >> following with us:
> >>
> >> 1. collect all the logs from all the nodes - clients and servers
> >> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
> >>
> >> --Yakov
> >>
> >> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> >>
> >>> Hi.
> >>>
> >>> There is also a few logs as :
> >>>
> >>> Failed to register marshalled class for more than 10 times in a row
> >>> (may affect performance).
> >>>
> >>> Can it be releated?
> >>>
> >>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com>
> wrote:
> >>>
> >>>> Hi all.
> >>>>
> >>>> We have 3 servers and cache configuration like:
> >>>>
> >>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
> >>>> name="DEFAULT">
> >>>> <property name="atomicityMode" value="ATOMIC" />
> >>>> <property name="cacheMode" value="PARTITIONED" />
> >>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
> >>>> <property name="backups" value="1" />
> >>>> <property name="affinity">
> >>>> <bean class="org.apache.ignite.cache
> >>>> .affinity.fair.FairAffinityFunction">
> >>>> <constructor-arg index="0" type="int"
> >>>> value="#{evamProperties['topology.cache.partition.size']}"/>
> >>>> </bean>
> >>>> </property>
> >>>> <!-- max memory is set to 0 means unlimited -->
> >>>> <property name="offHeapMaxMemory" value="0" />
> >>>> <property name="writeThrough" value="false" />
> >>>> <property name="writeBehindEnabled" value="false" />
> >>>> <property name="eagerTtl" value="false"/>
> >>>> </bean>
> >>>>
> >>>> For our worker threads we check heartbeat and if a thread did not sent
> >>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
> >>>> recreate it.
> >>>>
> >>>> As I can see all our worker threads are stucked in cache.put() state
> >>>> and interrupted and recreated regularly.
> >>>>
> >>>> What can be the reason we are stucked at put? Following is stacktrace
> >>>> for interruption error.
> >>>>
> >>>> javax.cache.CacheException: class org.apache.ignite.
> IgniteInterruptedException:
> >>>> Failed to wait for asynchronous operation permit (thread got
> interrupted).
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheUtils.c
> >>>> onvertToCacheException(GridCacheUtils.java:1502)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .cacheException(IgniteCacheProxy.java:2021)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1221)
> >>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.
> setSta
> >>>> te(ConfigManagerHelperEE.java:90)
> >>>> at com.intellica.project.helper.ee.
> StateMachineConfigManagerEEI
> >>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeCon
> >>>> fig(AbstractEventProcessor.java:175)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:130)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:80)
> >>>> at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine
> >>>> .java:199)
> >>>> at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHand
> >>>> ler.java:46)
> >>>> at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
> >>>> at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenario(AbstractScenarioWorker.java:130)
> >>>> at com.intellica.evam.engine.server.worker.AsyncWorker.
> processE
> >>>> vent(AsyncWorker.java:156)
> >>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> Asyn
> >>>> cWorker.java:88)
> >>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
> >>>> to wait for asynchronous operation permit (thread got interrupted).
> >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:747)
> >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:745)
> >>>> ... 14 more
> >>>> Caused by: java.lang.InterruptedException
> >>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquir
> >>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> >>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .putAsync(GridCacheAdapter.java:2541)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .put(GridCacheAdapter.java:2215)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1214)
> >>>> ... 11 more
> >>>>
> >>>>
> >>>> --
> >>>> Alper Tekinalp
> >>>>
> >>>> Software Developer
> >>>> Evam Streaming Analytics
> >>>>
> >>>> Atatürk Mah. Turgut Özal Bulv.
> >>>> Gardenya 5 Plaza K:6 Ataşehir
> >>>> 34758 İSTANBUL
> >>>>
> >>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> >>>> www.evam.com.tr
> >>>> <http://www.evam.com>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alper Tekinalp
> >>>
> >>> Software Developer
> >>> Evam Streaming Analytics
> >>>
> >>> Atatürk Mah. Turgut Özal Bulv.
> >>> Gardenya 5 Plaza K:6 Ataşehir
> >>> 34758 İSTANBUL
> >>>
> >>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> >>> www.evam.com.tr
> >>> <http://www.evam.com>
> >>>
> >>
> >>
> >
> >
> > --
> > Alper Tekinalp
> >
> > Software Developer
> > Evam Streaming Analytics
> >
> > Atatürk Mah. Turgut Özal Bulv.
> > Gardenya 5 Plaza K:6 Ataşehir
> > 34758 İSTANBUL
> >
> > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> > www.evam.com.tr
> > <http://www.evam.com>
> >
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
After restarting servers, that issue did not occur again. I guess for some
reason the cluster did not start normally.
Thanks responding so quick.
Regards.
On Tue, Oct 25, 2016 at 11:31 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> Alper, thanks for clarification, this will definitely help after we get the
> info I requested. This is the only way to go with the investigation.
>
> --Yakov
>
> 2016-10-25 11:20 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>
> > Hi Yakov.
> >
> > I should also mention that we load cache data from one server and wait
> the
> > data to be replicated to others. Can that cause such a situation, too?
> >
> > On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org>
> > wrote:
> >
> >> Alper,
> >>
> >> There can be multiple reasons.
> >>
> >> Can you please reproduce the issue one more time, collect and share the
> >> following with us:
> >>
> >> 1. collect all the logs from all the nodes - clients and servers
> >> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
> >>
> >> --Yakov
> >>
> >> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> >>
> >>> Hi.
> >>>
> >>> There is also a few logs as :
> >>>
> >>> Failed to register marshalled class for more than 10 times in a row
> >>> (may affect performance).
> >>>
> >>> Can it be releated?
> >>>
> >>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com>
> wrote:
> >>>
> >>>> Hi all.
> >>>>
> >>>> We have 3 servers and cache configuration like:
> >>>>
> >>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
> >>>> name="DEFAULT">
> >>>> <property name="atomicityMode" value="ATOMIC" />
> >>>> <property name="cacheMode" value="PARTITIONED" />
> >>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
> >>>> <property name="backups" value="1" />
> >>>> <property name="affinity">
> >>>> <bean class="org.apache.ignite.cache
> >>>> .affinity.fair.FairAffinityFunction">
> >>>> <constructor-arg index="0" type="int"
> >>>> value="#{evamProperties['topology.cache.partition.size']}"/>
> >>>> </bean>
> >>>> </property>
> >>>> <!-- max memory is set to 0 means unlimited -->
> >>>> <property name="offHeapMaxMemory" value="0" />
> >>>> <property name="writeThrough" value="false" />
> >>>> <property name="writeBehindEnabled" value="false" />
> >>>> <property name="eagerTtl" value="false"/>
> >>>> </bean>
> >>>>
> >>>> For our worker threads we check heartbeat and if a thread did not sent
> >>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
> >>>> recreate it.
> >>>>
> >>>> As I can see all our worker threads are stucked in cache.put() state
> >>>> and interrupted and recreated regularly.
> >>>>
> >>>> What can be the reason we are stucked at put? Following is stacktrace
> >>>> for interruption error.
> >>>>
> >>>> javax.cache.CacheException: class org.apache.ignite.
> IgniteInterruptedException:
> >>>> Failed to wait for asynchronous operation permit (thread got
> interrupted).
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheUtils.c
> >>>> onvertToCacheException(GridCacheUtils.java:1502)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .cacheException(IgniteCacheProxy.java:2021)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1221)
> >>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.
> setSta
> >>>> te(ConfigManagerHelperEE.java:90)
> >>>> at com.intellica.project.helper.ee.
> StateMachineConfigManagerEEI
> >>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeCon
> >>>> fig(AbstractEventProcessor.java:175)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:130)
> >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> process(
> >>>> AbstractEventProcessor.java:80)
> >>>> at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine
> >>>> .java:199)
> >>>> at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHand
> >>>> ler.java:46)
> >>>> at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
> >>>> at com.intellica.evam.engine.server.worker.
> AbstractScenarioWork
> >>>> er.runScenario(AbstractScenarioWorker.java:130)
> >>>> at com.intellica.evam.engine.server.worker.AsyncWorker.
> processE
> >>>> vent(AsyncWorker.java:156)
> >>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> Asyn
> >>>> cWorker.java:88)
> >>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
> >>>> to wait for asynchronous operation permit (thread got interrupted).
> >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:747)
> >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUt
> >>>> ils.java:745)
> >>>> ... 14 more
> >>>> Caused by: java.lang.InterruptedException
> >>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquir
> >>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> >>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .putAsync(GridCacheAdapter.java:2541)
> >>>> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> >>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> >>>> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter
> >>>> .put(GridCacheAdapter.java:2215)
> >>>> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy
> >>>> .put(IgniteCacheProxy.java:1214)
> >>>> ... 11 more
> >>>>
> >>>>
> >>>> --
> >>>> Alper Tekinalp
> >>>>
> >>>> Software Developer
> >>>> Evam Streaming Analytics
> >>>>
> >>>> Atatürk Mah. Turgut Özal Bulv.
> >>>> Gardenya 5 Plaza K:6 Ataşehir
> >>>> 34758 İSTANBUL
> >>>>
> >>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> >>>> www.evam.com.tr
> >>>> <http://www.evam.com>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alper Tekinalp
> >>>
> >>> Software Developer
> >>> Evam Streaming Analytics
> >>>
> >>> Atatürk Mah. Turgut Özal Bulv.
> >>> Gardenya 5 Plaza K:6 Ataşehir
> >>> 34758 İSTANBUL
> >>>
> >>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> >>> www.evam.com.tr
> >>> <http://www.evam.com>
> >>>
> >>
> >>
> >
> >
> > --
> > Alper Tekinalp
> >
> > Software Developer
> > Evam Streaming Analytics
> >
> > Atatürk Mah. Turgut Özal Bulv.
> > Gardenya 5 Plaza K:6 Ataşehir
> > 34758 İSTANBUL
> >
> > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> > www.evam.com.tr
> > <http://www.evam.com>
> >
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Yakov Zhdanov <yz...@apache.org>.
Alper, thanks for clarification, this will definitely help after we get the
info I requested. This is the only way to go with the investigation.
--Yakov
2016-10-25 11:20 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> Hi Yakov.
>
> I should also mention that we load cache data from one server and wait the
> data to be replicated to others. Can that cause such a situation, too?
>
> On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org>
> wrote:
>
>> Alper,
>>
>> There can be multiple reasons.
>>
>> Can you please reproduce the issue one more time, collect and share the
>> following with us:
>>
>> 1. collect all the logs from all the nodes - clients and servers
>> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>>
>> --Yakov
>>
>> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>>
>>> Hi.
>>>
>>> There is also a few logs as :
>>>
>>> Failed to register marshalled class for more than 10 times in a row
>>> (may affect performance).
>>>
>>> Can it be releated?
>>>
>>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We have 3 servers and cache configuration like:
>>>>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>>> name="DEFAULT">
>>>> <property name="atomicityMode" value="ATOMIC" />
>>>> <property name="cacheMode" value="PARTITIONED" />
>>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>> <property name="backups" value="1" />
>>>> <property name="affinity">
>>>> <bean class="org.apache.ignite.cache
>>>> .affinity.fair.FairAffinityFunction">
>>>> <constructor-arg index="0" type="int"
>>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>> </bean>
>>>> </property>
>>>> <!-- max memory is set to 0 means unlimited -->
>>>> <property name="offHeapMaxMemory" value="0" />
>>>> <property name="writeThrough" value="false" />
>>>> <property name="writeBehindEnabled" value="false" />
>>>> <property name="eagerTtl" value="false"/>
>>>> </bean>
>>>>
>>>> For our worker threads we check heartbeat and if a thread did not sent
>>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>>> recreate it.
>>>>
>>>> As I can see all our worker threads are stucked in cache.put() state
>>>> and interrupted and recreated regularly.
>>>>
>>>> What can be the reason we are stucked at put? Following is stacktrace
>>>> for interruption error.
>>>>
>>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .cacheException(IgniteCacheProxy.java:2021)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1221)
>>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>>> te(ConfigManagerHelperEE.java:90)
>>>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>>> fig(AbstractEventProcessor.java:175)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:130)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:80)
>>>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>>> .java:199)
>>>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>>> ler.java:46)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>>> vent(AsyncWorker.java:156)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>>> cWorker.java:88)
>>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>>> to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:747)
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:745)
>>>> ... 14 more
>>>> Caused by: java.lang.InterruptedException
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .putAsync(GridCacheAdapter.java:2541)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .put(GridCacheAdapter.java:2215)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1214)
>>>> ... 11 more
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
Re: Threads got stuck
Posted by Yakov Zhdanov <yz...@apache.org>.
Alper, thanks for clarification, this will definitely help after we get the
info I requested. This is the only way to go with the investigation.
--Yakov
2016-10-25 11:20 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> Hi Yakov.
>
> I should also mention that we load cache data from one server and wait the
> data to be replicated to others. Can that cause such a situation, too?
>
> On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org>
> wrote:
>
>> Alper,
>>
>> There can be multiple reasons.
>>
>> Can you please reproduce the issue one more time, collect and share the
>> following with us:
>>
>> 1. collect all the logs from all the nodes - clients and servers
>> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>>
>> --Yakov
>>
>> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>>
>>> Hi.
>>>
>>> There is also a few logs as :
>>>
>>> Failed to register marshalled class for more than 10 times in a row
>>> (may affect performance).
>>>
>>> Can it be releated?
>>>
>>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We have 3 servers and cache configuration like:
>>>>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>>> name="DEFAULT">
>>>> <property name="atomicityMode" value="ATOMIC" />
>>>> <property name="cacheMode" value="PARTITIONED" />
>>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>> <property name="backups" value="1" />
>>>> <property name="affinity">
>>>> <bean class="org.apache.ignite.cache
>>>> .affinity.fair.FairAffinityFunction">
>>>> <constructor-arg index="0" type="int"
>>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>> </bean>
>>>> </property>
>>>> <!-- max memory is set to 0 means unlimited -->
>>>> <property name="offHeapMaxMemory" value="0" />
>>>> <property name="writeThrough" value="false" />
>>>> <property name="writeBehindEnabled" value="false" />
>>>> <property name="eagerTtl" value="false"/>
>>>> </bean>
>>>>
>>>> For our worker threads we check heartbeat and if a thread did not sent
>>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>>> recreate it.
>>>>
>>>> As I can see all our worker threads are stucked in cache.put() state
>>>> and interrupted and recreated regularly.
>>>>
>>>> What can be the reason we are stucked at put? Following is stacktrace
>>>> for interruption error.
>>>>
>>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .cacheException(IgniteCacheProxy.java:2021)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1221)
>>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>>> te(ConfigManagerHelperEE.java:90)
>>>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>>> fig(AbstractEventProcessor.java:175)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:130)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:80)
>>>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>>> .java:199)
>>>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>>> ler.java:46)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>>> vent(AsyncWorker.java:156)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>>> cWorker.java:88)
>>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>>> to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:747)
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:745)
>>>> ... 14 more
>>>> Caused by: java.lang.InterruptedException
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .putAsync(GridCacheAdapter.java:2541)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .put(GridCacheAdapter.java:2215)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1214)
>>>> ... 11 more
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi Yakov.
I should also mention that we load cache data from one server and wait the
data to be replicated to others. Can that cause such a situation, too?
On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> Alper,
>
> There can be multiple reasons.
>
> Can you please reproduce the issue one more time, collect and share the
> following with us:
>
> 1. collect all the logs from all the nodes - clients and servers
> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>
> --Yakov
>
> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>
>> Hi.
>>
>> There is also a few logs as :
>>
>> Failed to register marshalled class for more than 10 times in a row (may
>> affect performance).
>>
>> Can it be releated?
>>
>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi all.
>>>
>>> We have 3 servers and cache configuration like:
>>>
>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>> name="DEFAULT">
>>> <property name="atomicityMode" value="ATOMIC" />
>>> <property name="cacheMode" value="PARTITIONED" />
>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>>> <property name="backups" value="1" />
>>> <property name="affinity">
>>> <bean class="org.apache.ignite.cache
>>> .affinity.fair.FairAffinityFunction">
>>> <constructor-arg index="0" type="int"
>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>> </bean>
>>> </property>
>>> <!-- max memory is set to 0 means unlimited -->
>>> <property name="offHeapMaxMemory" value="0" />
>>> <property name="writeThrough" value="false" />
>>> <property name="writeBehindEnabled" value="false" />
>>> <property name="eagerTtl" value="false"/>
>>> </bean>
>>>
>>> For our worker threads we check heartbeat and if a thread did not sent
>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>> recreate it.
>>>
>>> As I can see all our worker threads are stucked in cache.put() state and
>>> interrupted and recreated regularly.
>>>
>>> What can be the reason we are stucked at put? Following is stacktrace
>>> for interruption error.
>>>
>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>> onvertToCacheException(GridCacheUtils.java:1502)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .cacheException(IgniteCacheProxy.java:2021)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1221)
>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>> te(ConfigManagerHelperEE.java:90)
>>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>> fig(AbstractEventProcessor.java:175)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:130)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:80)
>>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>> .java:199)
>>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>> ler.java:46)
>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenario(AbstractScenarioWorker.java:130)
>>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>> vent(AsyncWorker.java:156)
>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>> cWorker.java:88)
>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>> to wait for asynchronous operation permit (thread got interrupted).
>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:747)
>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:745)
>>> ... 14 more
>>> Caused by: java.lang.InterruptedException
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .putAsync(GridCacheAdapter.java:2541)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .put(GridCacheAdapter.java:2215)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1214)
>>> ... 11 more
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi Yakov.
I should also mention that we load cache data from one server and wait the
data to be replicated to others. Can that cause such a situation, too?
On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> Alper,
>
> There can be multiple reasons.
>
> Can you please reproduce the issue one more time, collect and share the
> following with us:
>
> 1. collect all the logs from all the nodes - clients and servers
> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>
> --Yakov
>
> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>
>> Hi.
>>
>> There is also a few logs as :
>>
>> Failed to register marshalled class for more than 10 times in a row (may
>> affect performance).
>>
>> Can it be releated?
>>
>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi all.
>>>
>>> We have 3 servers and cache configuration like:
>>>
>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>> name="DEFAULT">
>>> <property name="atomicityMode" value="ATOMIC" />
>>> <property name="cacheMode" value="PARTITIONED" />
>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>>> <property name="backups" value="1" />
>>> <property name="affinity">
>>> <bean class="org.apache.ignite.cache
>>> .affinity.fair.FairAffinityFunction">
>>> <constructor-arg index="0" type="int"
>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>> </bean>
>>> </property>
>>> <!-- max memory is set to 0 means unlimited -->
>>> <property name="offHeapMaxMemory" value="0" />
>>> <property name="writeThrough" value="false" />
>>> <property name="writeBehindEnabled" value="false" />
>>> <property name="eagerTtl" value="false"/>
>>> </bean>
>>>
>>> For our worker threads we check heartbeat and if a thread did not sent
>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>> recreate it.
>>>
>>> As I can see all our worker threads are stucked in cache.put() state and
>>> interrupted and recreated regularly.
>>>
>>> What can be the reason we are stucked at put? Following is stacktrace
>>> for interruption error.
>>>
>>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>> onvertToCacheException(GridCacheUtils.java:1502)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .cacheException(IgniteCacheProxy.java:2021)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1221)
>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>> te(ConfigManagerHelperEE.java:90)
>>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>> fig(AbstractEventProcessor.java:175)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:130)
>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>> AbstractEventProcessor.java:80)
>>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>> .java:199)
>>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>> ler.java:46)
>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>> er.runScenario(AbstractScenarioWorker.java:130)
>>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>> vent(AsyncWorker.java:156)
>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>> cWorker.java:88)
>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>> to wait for asynchronous operation permit (thread got interrupted).
>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:747)
>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>> ils.java:745)
>>> ... 14 more
>>> Caused by: java.lang.InterruptedException
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .putAsync(GridCacheAdapter.java:2541)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .put(GridCacheAdapter.java:2215)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .put(IgniteCacheProxy.java:1214)
>>> ... 11 more
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Yakov Zhdanov <yz...@apache.org>.
Alper,
There can be multiple reasons.
Can you please reproduce the issue one more time, collect and share the
following with us:
1. collect all the logs from all the nodes - clients and servers
2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
--Yakov
2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> Hi.
>
> There is also a few logs as :
>
> Failed to register marshalled class for more than 10 times in a row (may
> affect performance).
>
> Can it be releated?
>
> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi all.
>>
>> We have 3 servers and cache configuration like:
>>
>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>> name="DEFAULT">
>> <property name="atomicityMode" value="ATOMIC" />
>> <property name="cacheMode" value="PARTITIONED" />
>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>> <property name="backups" value="1" />
>> <property name="affinity">
>> <bean class="org.apache.ignite.cache
>> .affinity.fair.FairAffinityFunction">
>> <constructor-arg index="0" type="int"
>> value="#{evamProperties['topology.cache.partition.size']}"/>
>> </bean>
>> </property>
>> <!-- max memory is set to 0 means unlimited -->
>> <property name="offHeapMaxMemory" value="0" />
>> <property name="writeThrough" value="false" />
>> <property name="writeBehindEnabled" value="false" />
>> <property name="eagerTtl" value="false"/>
>> </bean>
>>
>> For our worker threads we check heartbeat and if a thread did not sent
>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>> recreate it.
>>
>> As I can see all our worker threads are stucked in cache.put() state and
>> interrupted and recreated regularly.
>>
>> What can be the reason we are stucked at put? Following is stacktrace for
>> interruption error.
>>
>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>> Failed to wait for asynchronous operation permit (thread got interrupted).
>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>> onvertToCacheException(GridCacheUtils.java:1502)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .cacheException(IgniteCacheProxy.java:2021)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1221)
>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>> te(ConfigManagerHelperEE.java:90)
>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>> fig(AbstractEventProcessor.java:175)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:130)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:80)
>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>> .java:199)
>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>> ler.java:46)
>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenario(AbstractScenarioWorker.java:130)
>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>> vent(AsyncWorker.java:156)
>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>> cWorker.java:88)
>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
>> wait for asynchronous operation permit (thread got interrupted).
>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:747)
>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:745)
>> ... 14 more
>> Caused by: java.lang.InterruptedException
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .putAsync(GridCacheAdapter.java:2541)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .put(GridCacheAdapter.java:2215)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1214)
>> ... 11 more
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
Re: Threads got stuck
Posted by Yakov Zhdanov <yz...@apache.org>.
Alper,
There can be multiple reasons.
Can you please reproduce the issue one more time, collect and share the
following with us:
1. collect all the logs from all the nodes - clients and servers
2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
--Yakov
2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> Hi.
>
> There is also a few logs as :
>
> Failed to register marshalled class for more than 10 times in a row (may
> affect performance).
>
> Can it be releated?
>
> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi all.
>>
>> We have 3 servers and cache configuration like:
>>
>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>> name="DEFAULT">
>> <property name="atomicityMode" value="ATOMIC" />
>> <property name="cacheMode" value="PARTITIONED" />
>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>> <property name="backups" value="1" />
>> <property name="affinity">
>> <bean class="org.apache.ignite.cache
>> .affinity.fair.FairAffinityFunction">
>> <constructor-arg index="0" type="int"
>> value="#{evamProperties['topology.cache.partition.size']}"/>
>> </bean>
>> </property>
>> <!-- max memory is set to 0 means unlimited -->
>> <property name="offHeapMaxMemory" value="0" />
>> <property name="writeThrough" value="false" />
>> <property name="writeBehindEnabled" value="false" />
>> <property name="eagerTtl" value="false"/>
>> </bean>
>>
>> For our worker threads we check heartbeat and if a thread did not sent
>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>> recreate it.
>>
>> As I can see all our worker threads are stucked in cache.put() state and
>> interrupted and recreated regularly.
>>
>> What can be the reason we are stucked at put? Following is stacktrace for
>> interruption error.
>>
>> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
>> Failed to wait for asynchronous operation permit (thread got interrupted).
>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>> onvertToCacheException(GridCacheUtils.java:1502)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .cacheException(IgniteCacheProxy.java:2021)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1221)
>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>> te(ConfigManagerHelperEE.java:90)
>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>> fig(AbstractEventProcessor.java:175)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:130)
>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>> AbstractEventProcessor.java:80)
>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>> .java:199)
>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>> ler.java:46)
>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>> er.runScenario(AbstractScenarioWorker.java:130)
>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>> vent(AsyncWorker.java:156)
>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>> cWorker.java:88)
>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
>> wait for asynchronous operation permit (thread got interrupted).
>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:747)
>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>> ils.java:745)
>> ... 14 more
>> Caused by: java.lang.InterruptedException
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .putAsync(GridCacheAdapter.java:2541)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .put(GridCacheAdapter.java:2215)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .put(IgniteCacheProxy.java:1214)
>> ... 11 more
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
There is also a few logs as :
Failed to register marshalled class for more than 10 times in a row (may
affect performance).
Can it be releated?
On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
> Hi all.
>
> We have 3 servers and cache configuration like:
>
> <bean class="org.apache.ignite.configuration.CacheConfiguration"
> name="DEFAULT">
> <property name="atomicityMode" value="ATOMIC" />
> <property name="cacheMode" value="PARTITIONED" />
> <property name="memoryMode" value="OFFHEAP_TIERED" />
> <property name="backups" value="1" />
> <property name="affinity">
> <bean class="org.apache.ignite.cache.affinity.fair.
> FairAffinityFunction">
> <constructor-arg index="0" type="int"
> value="#{evamProperties['topology.cache.partition.size']}"/>
> </bean>
> </property>
> <!-- max memory is set to 0 means unlimited -->
> <property name="offHeapMaxMemory" value="0" />
> <property name="writeThrough" value="false" />
> <property name="writeBehindEnabled" value="false" />
> <property name="eagerTtl" value="false"/>
> </bean>
>
> For our worker threads we check heartbeat and if a thread did not sent
> heart beat for 10 minutes we consider it as stucked and interrrupt and
> recreate it.
>
> As I can see all our worker threads are stucked in cache.put() state and
> interrupted and recreated regularly.
>
> What can be the reason we are stucked at put? Following is stacktrace for
> interruption error.
>
> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
> Failed to wait for asynchronous operation permit (thread got interrupted).
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1502)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2021)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(
> ConfigManagerHelperEE.java:90)
> at com.intellica.project.helper.ee.StateMachineConfigManagerEEImp
> l.store(StateMachineConfigManagerEEImpl.java:53)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeConfig(AbstractEventProcessor.java:175)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:130)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:80)
> at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine.java:199)
> at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHandler.java:46)
> at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenarioLogic(AbstractScenarioWorker.java:172)
> at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenario(AbstractScenarioWorker.java:130)
> at com.intellica.evam.engine.server.worker.AsyncWorker.
> processEvent(AsyncWorker.java:156)
> at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> AsyncWorker.java:88)
> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
> wait for asynchronous operation permit (thread got interrupted).
> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:747)
> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:745)
> ... 14 more
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.
> asyncOpAcquire(GridCacheAdapter.java:4597)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.put(GridCacheAdapter.java:2215)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
> ... 11 more
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Threads got stuck
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
There is also a few logs as :
Failed to register marshalled class for more than 10 times in a row (may
affect performance).
Can it be releated?
On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
> Hi all.
>
> We have 3 servers and cache configuration like:
>
> <bean class="org.apache.ignite.configuration.CacheConfiguration"
> name="DEFAULT">
> <property name="atomicityMode" value="ATOMIC" />
> <property name="cacheMode" value="PARTITIONED" />
> <property name="memoryMode" value="OFFHEAP_TIERED" />
> <property name="backups" value="1" />
> <property name="affinity">
> <bean class="org.apache.ignite.cache.affinity.fair.
> FairAffinityFunction">
> <constructor-arg index="0" type="int"
> value="#{evamProperties['topology.cache.partition.size']}"/>
> </bean>
> </property>
> <!-- max memory is set to 0 means unlimited -->
> <property name="offHeapMaxMemory" value="0" />
> <property name="writeThrough" value="false" />
> <property name="writeBehindEnabled" value="false" />
> <property name="eagerTtl" value="false"/>
> </bean>
>
> For our worker threads we check heartbeat and if a thread did not sent
> heart beat for 10 minutes we consider it as stucked and interrrupt and
> recreate it.
>
> As I can see all our worker threads are stucked in cache.put() state and
> interrupted and recreated regularly.
>
> What can be the reason we are stucked at put? Following is stacktrace for
> interruption error.
>
> javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException:
> Failed to wait for asynchronous operation permit (thread got interrupted).
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> convertToCacheException(GridCacheUtils.java:1502)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.
> cacheException(IgniteCacheProxy.java:2021)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(
> ConfigManagerHelperEE.java:90)
> at com.intellica.project.helper.ee.StateMachineConfigManagerEEImp
> l.store(StateMachineConfigManagerEEImpl.java:53)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.
> storeConfig(AbstractEventProcessor.java:175)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:130)
> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
> AbstractEventProcessor.java:80)
> at com.evelopers.unimod.runtime.ModelEngine.process(
> ModelEngine.java:199)
> at com.evelopers.unimod.runtime.StrictHandler.handle(
> StrictHandler.java:46)
> at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenarioLogic(AbstractScenarioWorker.java:172)
> at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.
> runScenario(AbstractScenarioWorker.java:130)
> at com.intellica.evam.engine.server.worker.AsyncWorker.
> processEvent(AsyncWorker.java:156)
> at com.intellica.evam.engine.server.worker.AsyncWorker.run(
> AsyncWorker.java:88)
> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
> wait for asynchronous operation permit (thread got interrupted).
> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:747)
> at org.apache.ignite.internal.util.IgniteUtils$2.apply(
> IgniteUtils.java:745)
> ... 14 more
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.
> asyncOpAcquire(GridCacheAdapter.java:4597)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
> at org.apache.ignite.internal.processors.cache.
> GridCacheAdapter.put(GridCacheAdapter.java:2215)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
> ... 11 more
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>