You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Alper Tekinalp <al...@evam.com> on 2017/03/20 09:57:07 UTC
Client got stucked on get operation
Hi all.
We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
connects eachother as server but connects server 1 as client. In a point of
the time server 3 got stucked at:
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4665)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1388)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:1121)
at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.evam.cache.client.CachePassthroughInvocationHandler.invokeInternal(CachePassthroughInvocationHandler.java:99)
at
com.evam.cache.client.CachePassthroughInvocationHandler.invoke(CachePassthroughInvocationHandler.java:78)
at com.sun.proxy.$Proxy56.get(Unknown Source)
when getting record from server 1. Long after that server 2 also got
stucked at the same trace and also server 2 and server 3 disconnects from
each other.
We investigated gc logs and there is not an unusual behaviour. One things
is server 1 logs errors as follows when server 2 and server 3 disconnects:
[ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg -
Received message without registered handler (will ignore)
[msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
[arr= true, ctx=false, start=0], partId=199, flags=1,
topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0, createTtl=0,
accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
cacheDesc=null]
Registered listeners:
Where should we look for the main cause of the problem? What can cause such
a behaviour? There seems nothing wrong on server 1 logs etc.
We use ignite 1.8.3.
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
Looks weird. Seems cache descriptor was not initialized by some reason or
cache was destroyed unexpectedly.
Most possibly, there is a race and there is no way to recover.
On Wed, Apr 12, 2017 at 3:00 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi.
>
> We encountered the same situation again. That time we get the following
> error:
>
> 12/Apr/2017 10:08:17 ERROR 72832749 exchange-worker-#199%null%
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture(L:495) - Failed to reinitialize local
> partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=46, minorTopVer=45], nodeId=172e4264, evt=DISCOVERY_CUSTOM_EVT]
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> initStartedCacheOnCoordinator(CacheAffinitySharedManager.java:747)
> at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.
> onCacheChangeRequest(CacheAffinitySharedManager.java:413)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onCacheChangeRequest(
> GridDhtPartitionsExchangeFuture.java:571)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:454)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$ExchangeWorker.body(
> GridCachePartitionExchangeManager.java:1670)
> at org.apache.ignite.internal.util.worker.GridWorker.run(
> GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> 12/Apr/2017 10:08:18 ERROR 72833126 exchange-worker-#199%null%
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager(L:495)
> - Runtime error caught during grid runnable execution: GridWorker name=partition-exchanger,
> gridName=null, finished=false, isCancelled=false, hashCode=1466668876,
> interrupted=false, runner=exchange-worker-#199%null%
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.partitionMap(GridDhtPartitionTopologyImpl.
> java:973)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager.createPartitionsFullMessage(
> GridCachePartitionExchangeManager.java:855)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.createPartitionsMessage(
> GridDhtPartitionsExchangeFuture.java:966)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.sendAllPartitions(
> GridDhtPartitionsExchangeFuture.java:977)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.sendAllPartitions(
> GridDhtPartitionsExchangeFuture.java:1313)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFutur
> e.java:1142)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$7.apply(GridCachePartitionExchangeMana
> ger.java:1295)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$7.apply(GridCachePartitionExchangeMana
> ger.java:1292)
> at org.apache.ignite.internal.util.future.GridFutureAdapter$
> ArrayListener.apply(GridFutureAdapter.java:456)
> at org.apache.ignite.internal.util.future.GridFutureAdapter$
> ArrayListener.apply(GridFutureAdapter.java:439)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> notifyListener(GridFutureAdapter.java:271)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> notifyListeners(GridFutureAdapter.java:259)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> onDone(GridFutureAdapter.java:389)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> onDone(GridFutureAdapter.java:355)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFutur
> e.java:1053)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFutur
> e.java:88)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> onDone(GridFutureAdapter.java:343)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur
> e.java:512)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$ExchangeWorker.body(
> GridCachePartitionExchangeManager.java:1670)
> at org.apache.ignite.internal.util.worker.GridWorker.run(
> GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
>
> After that exchange-worker dies and all put and get operations blocks.
>
> 1 - What does above error mean? When does it happen?
> 2 - Is there a way to avoid or recover from that state?
>
> Regards.
>
> On Thu, Apr 6, 2017 at 5:50 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Alper,
>>
>> I see no starvation here. Looks like WriteBehindStore waits for its queue
>> has flushed.
>> Threaddump is needed to understand if Flusher threads also waits for smth.
>>
>> On Thu, Apr 6, 2017 at 4:40 PM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi Andrey.
>>>
>>> Ignite logs are at the attachment.
>>>
>>> Interruption exception is on 30/Mar/2017 15:56:03. starvation log is on
>>> the second node.
>>>
>>> I do not have threaddump. I can provide if the problem repeats.
>>>
>>> Thanks for your helps!
>>>
>>> On Thu, Apr 6, 2017 at 4:27 PM, Andrey Mashenkov <
>>> andrey.mashenkov@gmail.com> wrote:
>>>
>>>> Hi Alper,
>>>>
>>>> Would you please provide full treaddump and full log?
>>>>
>>>> On Thu, Apr 6, 2017 at 4:08 PM, nragon <nuno.goncalves@wedotechnologi
>>>> es.com> wrote:
>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Please note that the same flink job without ignite runs around 30k/s
>>>>> with
>>>>> ignite get method goes to 2k/s. If you don't mind taking a look at
>>>>> http://apache-ignite-users.70518.x6.nabble.com/Client-near-c
>>>>> ache-with-Apache-Flink-td11627.html.
>>>>> I only said it was related because of the thread dump Alper mentioned
>>>>> seemed
>>>>> alike and because his architecture with clients/servers are also very
>>>>> similiar to mine.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://apache-ignite-users.705
>>>>> 18.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313
>>>>> p11780.html
>>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
We encountered the same situation again. That time we get the following
error:
12/Apr/2017 10:08:17 ERROR 72832749 exchange-worker-#199%null%
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture(L:495)
- Failed to reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=46,
minorTopVer=45], nodeId=172e4264, evt=DISCOVERY_CUSTOM_EVT]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initStartedCacheOnCoordinator(CacheAffinitySharedManager.java:747)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:413)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:571)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:454)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1670)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
12/Apr/2017 10:08:18 ERROR 72833126 exchange-worker-#199%null%
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager(L:495)
- Runtime error caught during grid runnable execution: GridWorker
name=partition-exchanger,
gridName=null, finished=false, isCancelled=false, hashCode=1466668876,
interrupted=false, runner=exchange-worker-#199%null%
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.partitionMap(GridDhtPartitionTopologyImpl.java:973)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.createPartitionsFullMessage(GridCachePartitionExchangeManager.java:855)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createPartitionsMessage(GridDhtPartitionsExchangeFuture.java:966)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendAllPartitions(GridDhtPartitionsExchangeFuture.java:977)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendAllPartitions(GridDhtPartitionsExchangeFuture.java:1313)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1142)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$7.apply(GridCachePartitionExchangeManager.java:1295)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$7.apply(GridCachePartitionExchangeManager.java:1292)
at
org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:456)
at
org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:439)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1053)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:88)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:343)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:512)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1670)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
After that exchange-worker dies and all put and get operations blocks.
1 - What does above error mean? When does it happen?
2 - Is there a way to avoid or recover from that state?
Regards.
On Thu, Apr 6, 2017 at 5:50 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:
> Hi Alper,
>
> I see no starvation here. Looks like WriteBehindStore waits for its queue
> has flushed.
> Threaddump is needed to understand if Flusher threads also waits for smth.
>
> On Thu, Apr 6, 2017 at 4:40 PM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi Andrey.
>>
>> Ignite logs are at the attachment.
>>
>> Interruption exception is on 30/Mar/2017 15:56:03. starvation log is on
>> the second node.
>>
>> I do not have threaddump. I can provide if the problem repeats.
>>
>> Thanks for your helps!
>>
>> On Thu, Apr 6, 2017 at 4:27 PM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>
>>> Hi Alper,
>>>
>>> Would you please provide full treaddump and full log?
>>>
>>> On Thu, Apr 6, 2017 at 4:08 PM, nragon <nuno.goncalves@wedotechnologi
>>> es.com> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> Please note that the same flink job without ignite runs around 30k/s
>>>> with
>>>> ignite get method goes to 2k/s. If you don't mind taking a look at
>>>> http://apache-ignite-users.70518.x6.nabble.com/Client-near-c
>>>> ache-with-Apache-Flink-td11627.html.
>>>> I only said it was related because of the thread dump Alper mentioned
>>>> seemed
>>>> alike and because his architecture with clients/servers are also very
>>>> similiar to mine.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-ignite-users.705
>>>> 18.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11780.html
>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
I see no starvation here. Looks like WriteBehindStore waits for its queue
has flushed.
Threaddump is needed to understand if Flusher threads also waits for smth.
On Thu, Apr 6, 2017 at 4:40 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi Andrey.
>
> Ignite logs are at the attachment.
>
> Interruption exception is on 30/Mar/2017 15:56:03. starvation log is on
> the second node.
>
> I do not have threaddump. I can provide if the problem repeats.
>
> Thanks for your helps!
>
> On Thu, Apr 6, 2017 at 4:27 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Alper,
>>
>> Would you please provide full treaddump and full log?
>>
>> On Thu, Apr 6, 2017 at 4:08 PM, nragon <nuno.goncalves@wedotechnologi
>> es.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> Please note that the same flink job without ignite runs around 30k/s with
>>> ignite get method goes to 2k/s. If you don't mind taking a look at
>>> http://apache-ignite-users.70518.x6.nabble.com/Client-near-c
>>> ache-with-Apache-Flink-td11627.html.
>>> I only said it was related because of the thread dump Alper mentioned
>>> seemed
>>> alike and because his architecture with clients/servers are also very
>>> similiar to mine.
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.705
>>> 18.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11780.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi Andrey.
Ignite logs are at the attachment.
Interruption exception is on 30/Mar/2017 15:56:03. starvation log is on the
second node.
I do not have threaddump. I can provide if the problem repeats.
Thanks for your helps!
On Thu, Apr 6, 2017 at 4:27 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:
> Hi Alper,
>
> Would you please provide full treaddump and full log?
>
> On Thu, Apr 6, 2017 at 4:08 PM, nragon <nuno.goncalves@
> wedotechnologies.com> wrote:
>
>> Hi Andrew,
>>
>> Please note that the same flink job without ignite runs around 30k/s with
>> ignite get method goes to 2k/s. If you don't mind taking a look at
>> http://apache-ignite-users.70518.x6.nabble.com/Client-near-
>> cache-with-Apache-Flink-td11627.html.
>> I only said it was related because of the thread dump Alper mentioned
>> seemed
>> alike and because his architecture with clients/servers are also very
>> similiar to mine.
>>
>> Thanks
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11780.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
Would you please provide full treaddump and full log?
On Thu, Apr 6, 2017 at 4:08 PM, nragon <nu...@wedotechnologies.com>
wrote:
> Hi Andrew,
>
> Please note that the same flink job without ignite runs around 30k/s with
> ignite get method goes to 2k/s. If you don't mind taking a look at
> http://apache-ignite-users.70518.x6.nabble.com/Client-
> near-cache-with-Apache-Flink-td11627.html.
> I only said it was related because of the thread dump Alper mentioned
> seemed
> alike and because his architecture with clients/servers are also very
> similiar to mine.
>
> Thanks
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11780.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by nragon <nu...@wedotechnologies.com>.
Hi Andrew,
Please note that the same flink job without ignite runs around 30k/s with
ignite get method goes to 2k/s. If you don't mind taking a look at
http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-td11627.html.
I only said it was related because of the thread dump Alper mentioned seemed
alike and because his architecture with clients/servers are also very
similiar to mine.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11780.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
As I found from logs one of out threads are interrupted and had an
exception:
30/Mar/2017 15:56:03 ERROR 133548100 [DeploymentWorker-0]
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl(L:495)
- DataStreamer operation failed.
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:868)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:828)
at
org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:456)
at
org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:439)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1412)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:932)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addDataInternal(DataStreamerImpl.java:631)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:667)
After that error all other threads are stucked on the previously mentioned
state. Is that related? If so how? That interrupted thread and others
accessing different caches.
After 2 minutes later from that interruption there exist that log:
30/Mar/2017 15:57:25 WARN 133630286 [grid-timeout-worker-#127%null%]
org.apache.ignite.internal.util.typedef.G(L:480) - >>> Possible starvation
in striped pool: sys-stripe-30-#31%null%
[]
deadlock: false
completed: 1175974
Thread [name="sys-stripe-30-#31%null%", id=150, state=WAITING,
blockCnt=541, waitCnt=1079913]
Lock [object=java.lang.Object@38d4ca3, ownerName=null, ownerId=-1]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at o.a.i.i.util.worker.GridWorker.join(GridWorker.java:233)
at o.a.i.i.util.IgniteUtils.join(IgniteUtils.java:4631)
at
o.a.i.i.processors.cache.store.GridCacheWriteBehindStore.stop(GridCacheWriteBehindStore.java:352)
at
o.a.i.i.processors.cache.store.GridCacheStoreManagerAdapter.stop0(GridCacheStoreManagerAdapter.java:257)
at
o.a.i.i.processors.cache.GridCacheManagerAdapter.stop(GridCacheManagerAdapter.java:83)
at
o.a.i.i.processors.cache.GridCacheProcessor.stopCache(GridCacheProcessor.java:1138)
at
o.a.i.i.processors.cache.GridCacheProcessor.prepareCacheStop(GridCacheProcessor.java:1778)
at
o.a.i.i.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:1815)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1051)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:88)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:332)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1402)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$400(GridDhtPartitionsExchangeFuture.java:88)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$4.apply(GridDhtPartitionsExchangeFuture.java:1371)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$4.apply(GridDhtPartitionsExchangeFuture.java:1359)
at
o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at
o.a.i.i.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
at
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1359)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager.processFullPartitionUpdate(GridCachePartitionExchangeManager.java:1234)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager.access$1300(GridCachePartitionExchangeManager.java:116)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:317)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:315)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1987)
at
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1969)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1215)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:843)
at
o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
at
o.a.i.i.managers.communication.GridIoManager$6.run(GridIoManager.java:783)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
at java.lang.Thread.run(Thread.java:745)
Does that mean we have not enough sys- threads?
On Thu, Apr 6, 2017 at 3:09 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:
> Hi,
>
> Looks like not related.
> Most of time spent in non-ignite methods, but in flink serializer.
> Also I see a contention in flink Serializer.
>
> On Thu, Apr 6, 2017 at 2:27 PM, nragon <nuno.goncalves@
> wedotechnologies.com> wrote:
>
>> Hi,
>>
>> Might be related with:
>> http://apache-ignite-users.70518.x6.nabble.com/Client-near-
>> cache-with-Apache-Flink-td11627.html
>>
>> I've attached my jfr file along with some other metrics and a test class
>> with the same configurations.
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11775.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Looks like not related.
Most of time spent in non-ignite methods, but in flink serializer.
Also I see a contention in flink Serializer.
On Thu, Apr 6, 2017 at 2:27 PM, nragon <nu...@wedotechnologies.com>
wrote:
> Hi,
>
> Might be related with:
> http://apache-ignite-users.70518.x6.nabble.com/Client-
> near-cache-with-Apache-Flink-td11627.html
>
> I've attached my jfr file along with some other metrics and a test class
> with the same configurations.
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11775.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by nragon <nu...@wedotechnologies.com>.
Hi,
Might be related with:
http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-td11627.html
I've attached my jfr file along with some other metrics and a test class
with the same configurations.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-got-stucked-on-get-operation-tp11313p11775.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
So, you have standalone Ignite instance on "cache server" and grid of 2
instances on "app servers".
How are you access this standalone instance from "app servers"? With client
ignite instances? Would you please provide its config?
When you see some threads stucked on remote operation, the first steps
should be to check
- if there are ("sys-" or "pub-") threads making progress
- if "exchange-worker" thread and "disco-" threads either some do progress
or wait for tasks.
- check logs for explicit locks if transactional caches are used.
- find threads statistics in logs, and check if there is no starvation.
If all thread are waiting for futures and thread pools are not full
(especially system), then it looks like some message was missed due to some
bug.
On Thu, Apr 6, 2017 at 12:03 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi Andrey.
>
> You can find configuration files and basic structure in the attachment.
> App servers connects cache server as client programmatically so no
> configuration for that.
>
>
> On Tue, Apr 4, 2017 at 6:15 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Alper,
>>
>> Still not enough clear. Would you please share a grid configuration?
>>
>> On Tue, Apr 4, 2017 at 4:45 PM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi Andrey.
>>>
>>> The structure is a bit complicated. Let me try to explane: We have 2
>>> types of applications: 1 is cache server which contains external caches, 1
>>> is our application servers that runs the logic. Application servers also
>>> creates and maintains caches. We have 2 application servers that connects
>>> each other as servers. And these 2 application servers connects cache
>>> server as clients. Application servers and cache server are 2 different
>>> topologies.
>>>
>>> Sometime worker threads are stucked on:
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>> 0(GridFutureAdapter.java:161)
>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>> (GridFutureAdapter.java:119)
>>>
>>> on put or get operations either caches on application servers or while
>>> accessing caches on cache server.
>>>
>>> Our cached on application servers are affinity based so we only access
>>> records on local server.
>>>
>>>
>>>
>>>
>>> On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov <
>>> andrey.mashenkov@gmail.com> wrote:
>>>
>>>> Hi Alper,
>>>>
>>>> Thread is waiting for result of remote get operation.
>>>>
>>>> Are you sure your server 1 is standalone and has its own topology? I
>>>> can't understand, how 1 and 3 connects to 1 as client.
>>>> Would you please clarify?
>>>>
>>>> On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <al...@evam.com> wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> Can someone point me a direction that why a thread can stuck on the
>>>>> trace above? Where should I look for? How can I investicate the issue?
>>>>> Where should I look?
>>>>>
>>>>> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>>
>>>>>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and
>>>>>> 3 connects eachother as server but connects server 1 as client. In a point
>>>>>> of the time server 3 got stucked at:
>>>>>>
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java
>>>>>> :186)
>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>>> 0(GridFutureAdapter.java:161)
>>>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>>> (GridFutureAdapter.java:119)
>>>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>>>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>>>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>>> .get(GridCacheAdapter.java:4665)
>>>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>>> .get(GridCacheAdapter.java:1388)
>>>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>>>> .get(IgniteCacheProxy.java:1121)
>>>>>> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>>> thodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>>> keInternal(CachePassthroughInvocationHandler.java:99)
>>>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>>> ke(CachePassthroughInvocationHandler.java:78)
>>>>>> at com.sun.proxy.$Proxy56.get(Unknown Source)
>>>>>>
>>>>>> when getting record from server 1. Long after that server 2 also got
>>>>>> stucked at the same trace and also server 2 and server 3 disconnects from
>>>>>> each other.
>>>>>>
>>>>>> We investigated gc logs and there is not an unusual behaviour. One
>>>>>> things is server 1 logs errors as follows when server 2 and server 3
>>>>>> disconnects:
>>>>>>
>>>>>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%]
>>>>>> msg - Received message without registered handler (will ignore)
>>>>>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>>>>>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>>>>>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>>>>>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>>>>>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>>>>>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>>> cacheDesc=null]
>>>>>> Registered listeners:
>>>>>>
>>>>>>
>>>>>> Where should we look for the main cause of the problem? What can
>>>>>> cause such a behaviour? There seems nothing wrong on server 1 logs etc.
>>>>>>
>>>>>> We use ignite 1.8.3.
>>>>>>
>>>>>> --
>>>>>> Alper Tekinalp
>>>>>>
>>>>>> Software Developer
>>>>>> Evam Streaming Analytics
>>>>>>
>>>>>> Atatürk Mah. Turgut Özal Bulv.
>>>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>>>> 34758 İSTANBUL
>>>>>>
>>>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>>>> www.evam.com.tr
>>>>>> <http://www.evam.com>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Alper Tekinalp
>>>>>
>>>>> Software Developer
>>>>> Evam Streaming Analytics
>>>>>
>>>>> Atatürk Mah. Turgut Özal Bulv.
>>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>>> 34758 İSTANBUL
>>>>>
>>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>>> www.evam.com.tr
>>>>> <http://www.evam.com>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi Andrey.
You can find configuration files and basic structure in the attachment. App
servers connects cache server as client programmatically so no
configuration for that.
On Tue, Apr 4, 2017 at 6:15 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:
> Hi Alper,
>
> Still not enough clear. Would you please share a grid configuration?
>
> On Tue, Apr 4, 2017 at 4:45 PM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi Andrey.
>>
>> The structure is a bit complicated. Let me try to explane: We have 2
>> types of applications: 1 is cache server which contains external caches, 1
>> is our application servers that runs the logic. Application servers also
>> creates and maintains caches. We have 2 application servers that connects
>> each other as servers. And these 2 application servers connects cache
>> server as clients. Application servers and cache server are 2 different
>> topologies.
>>
>> Sometime worker threads are stucked on:
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> 0(GridFutureAdapter.java:161)
>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> (GridFutureAdapter.java:119)
>>
>> on put or get operations either caches on application servers or while
>> accessing caches on cache server.
>>
>> Our cached on application servers are affinity based so we only access
>> records on local server.
>>
>>
>>
>>
>> On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>
>>> Hi Alper,
>>>
>>> Thread is waiting for result of remote get operation.
>>>
>>> Are you sure your server 1 is standalone and has its own topology? I
>>> can't understand, how 1 and 3 connects to 1 as client.
>>> Would you please clarify?
>>>
>>> On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <al...@evam.com> wrote:
>>>
>>>> Hi.
>>>>
>>>> Can someone point me a direction that why a thread can stuck on the
>>>> trace above? Where should I look for? How can I investicate the issue?
>>>> Where should I look?
>>>>
>>>> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com>
>>>> wrote:
>>>>
>>>>> Hi all.
>>>>>
>>>>>
>>>>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
>>>>> connects eachother as server but connects server 1 as client. In a point of
>>>>> the time server 3 got stucked at:
>>>>>
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>> 0(GridFutureAdapter.java:161)
>>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>>> (GridFutureAdapter.java:119)
>>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>> .get(GridCacheAdapter.java:4665)
>>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>>> .get(GridCacheAdapter.java:1388)
>>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>>> .get(IgniteCacheProxy.java:1121)
>>>>> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>> keInternal(CachePassthroughInvocationHandler.java:99)
>>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>>> ke(CachePassthroughInvocationHandler.java:78)
>>>>> at com.sun.proxy.$Proxy56.get(Unknown Source)
>>>>>
>>>>> when getting record from server 1. Long after that server 2 also got
>>>>> stucked at the same trace and also server 2 and server 3 disconnects from
>>>>> each other.
>>>>>
>>>>> We investigated gc logs and there is not an unusual behaviour. One
>>>>> things is server 1 logs errors as follows when server 2 and server 3
>>>>> disconnects:
>>>>>
>>>>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg
>>>>> - Received message without registered handler (will ignore)
>>>>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>>>>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>>>>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>>>>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>>>>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>>>>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>>> cacheDesc=null]
>>>>> Registered listeners:
>>>>>
>>>>>
>>>>> Where should we look for the main cause of the problem? What can cause
>>>>> such a behaviour? There seems nothing wrong on server 1 logs etc.
>>>>>
>>>>> We use ignite 1.8.3.
>>>>>
>>>>> --
>>>>> Alper Tekinalp
>>>>>
>>>>> Software Developer
>>>>> Evam Streaming Analytics
>>>>>
>>>>> Atatürk Mah. Turgut Özal Bulv.
>>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>>> 34758 İSTANBUL
>>>>>
>>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>>> www.evam.com.tr
>>>>> <http://www.evam.com>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
Still not enough clear. Would you please share a grid configuration?
On Tue, Apr 4, 2017 at 4:45 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi Andrey.
>
> The structure is a bit complicated. Let me try to explane: We have 2 types
> of applications: 1 is cache server which contains external caches, 1 is our
> application servers that runs the logic. Application servers also creates
> and maintains caches. We have 2 application servers that connects each
> other as servers. And these 2 application servers connects cache server as
> clients. Application servers and cache server are 2 different topologies.
>
> Sometime worker threads are stucked on:
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
> 0(GridFutureAdapter.java:161)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
> (GridFutureAdapter.java:119)
>
> on put or get operations either caches on application servers or while
> accessing caches on cache server.
>
> Our cached on application servers are affinity based so we only access
> records on local server.
>
>
>
>
> On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Alper,
>>
>> Thread is waiting for result of remote get operation.
>>
>> Are you sure your server 1 is standalone and has its own topology? I
>> can't understand, how 1 and 3 connects to 1 as client.
>> Would you please clarify?
>>
>> On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi.
>>>
>>> Can someone point me a direction that why a thread can stuck on the
>>> trace above? Where should I look for? How can I investicate the issue?
>>> Where should I look?
>>>
>>> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>>
>>>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
>>>> connects eachother as server but connects server 1 as client. In a point of
>>>> the time server 3 got stucked at:
>>>>
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>> 0(GridFutureAdapter.java:161)
>>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>>> (GridFutureAdapter.java:119)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .get(GridCacheAdapter.java:4665)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .get(GridCacheAdapter.java:1388)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .get(IgniteCacheProxy.java:1121)
>>>> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>> keInternal(CachePassthroughInvocationHandler.java:99)
>>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>>> ke(CachePassthroughInvocationHandler.java:78)
>>>> at com.sun.proxy.$Proxy56.get(Unknown Source)
>>>>
>>>> when getting record from server 1. Long after that server 2 also got
>>>> stucked at the same trace and also server 2 and server 3 disconnects from
>>>> each other.
>>>>
>>>> We investigated gc logs and there is not an unusual behaviour. One
>>>> things is server 1 logs errors as follows when server 2 and server 3
>>>> disconnects:
>>>>
>>>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg
>>>> - Received message without registered handler (will ignore)
>>>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>>>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>>>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>>>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>>>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>>>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>>> cacheDesc=null]
>>>> Registered listeners:
>>>>
>>>>
>>>> Where should we look for the main cause of the problem? What can cause
>>>> such a behaviour? There seems nothing wrong on server 1 logs etc.
>>>>
>>>> We use ignite 1.8.3.
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi Andrey.
The structure is a bit complicated. Let me try to explane: We have 2 types
of applications: 1 is cache server which contains external caches, 1 is our
application servers that runs the logic. Application servers also creates
and maintains caches. We have 2 application servers that connects each
other as servers. And these 2 application servers connects cache server as
clients. Application servers and cache server are 2 different topologies.
Sometime worker threads are stucked on:
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.
parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.
doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.
acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at org.apache.ignite.internal.util.future.GridFutureAdapter.
get0(GridFutureAdapter.java:161)
at org.apache.ignite.internal.util.future.GridFutureAdapter.
get(GridFutureAdapter.java:119)
on put or get operations either caches on application servers or while
accessing caches on cache server.
Our cached on application servers are affinity based so we only access
records on local server.
On Tue, Apr 4, 2017 at 4:09 PM, Andrey Mashenkov <andrey.mashenkov@gmail.com
> wrote:
> Hi Alper,
>
> Thread is waiting for result of remote get operation.
>
> Are you sure your server 1 is standalone and has its own topology? I can't
> understand, how 1 and 3 connects to 1 as client.
> Would you please clarify?
>
> On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi.
>>
>> Can someone point me a direction that why a thread can stuck on the trace
>> above? Where should I look for? How can I investicate the issue? Where
>> should I look?
>>
>> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com> wrote:
>>
>>> Hi all.
>>>
>>>
>>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
>>> connects eachother as server but connects server 1 as client. In a point of
>>> the time server 3 got stucked at:
>>>
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>> 0(GridFutureAdapter.java:161)
>>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>>> (GridFutureAdapter.java:119)
>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .get(GridCacheAdapter.java:4665)
>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>> .get(GridCacheAdapter.java:1388)
>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>> .get(IgniteCacheProxy.java:1121)
>>> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>> keInternal(CachePassthroughInvocationHandler.java:99)
>>> at com.evam.cache.client.CachePassthroughInvocationHandler.invo
>>> ke(CachePassthroughInvocationHandler.java:78)
>>> at com.sun.proxy.$Proxy56.get(Unknown Source)
>>>
>>> when getting record from server 1. Long after that server 2 also got
>>> stucked at the same trace and also server 2 and server 3 disconnects from
>>> each other.
>>>
>>> We investigated gc logs and there is not an unusual behaviour. One
>>> things is server 1 logs errors as follows when server 2 and server 3
>>> disconnects:
>>>
>>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg -
>>> Received message without registered handler (will ignore)
>>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>>> cacheDesc=null]
>>> Registered listeners:
>>>
>>>
>>> Where should we look for the main cause of the problem? What can cause
>>> such a behaviour? There seems nothing wrong on server 1 logs etc.
>>>
>>> We use ignite 1.8.3.
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Alper,
Thread is waiting for result of remote get operation.
Are you sure your server 1 is standalone and has its own topology? I can't
understand, how 1 and 3 connects to 1 as client.
Would you please clarify?
On Tue, Apr 4, 2017 at 3:43 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi.
>
> Can someone point me a direction that why a thread can stuck on the trace
> above? Where should I look for? How can I investicate the issue? Where
> should I look?
>
> On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com> wrote:
>
>> Hi all.
>>
>>
>> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
>> connects eachother as server but connects server 1 as client. In a point of
>> the time server 3 got stucked at:
>>
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> 0(GridFutureAdapter.java:161)
>> at org.apache.ignite.internal.util.future.GridFutureAdapter.get
>> (GridFutureAdapter.java:119)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .get(GridCacheAdapter.java:4665)
>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>> .get(GridCacheAdapter.java:1388)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>> .get(IgniteCacheProxy.java:1121)
>> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.evam.cache.client.CachePassthroughInvocationHandler.
>> invokeInternal(CachePassthroughInvocationHandler.java:99)
>> at com.evam.cache.client.CachePassthroughInvocationHandler.
>> invoke(CachePassthroughInvocationHandler.java:78)
>> at com.sun.proxy.$Proxy56.get(Unknown Source)
>>
>> when getting record from server 1. Long after that server 2 also got
>> stucked at the same trace and also server 2 and server 3 disconnects from
>> each other.
>>
>> We investigated gc logs and there is not an unusual behaviour. One things
>> is server 1 logs errors as follows when server 2 and server 3 disconnects:
>>
>> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg -
>> Received message without registered handler (will ignore)
>> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
>> [arr= true, ctx=false, start=0], partId=199, flags=1,
>> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0,
>> createTtl=0, accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a,
>> locTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=2937],
>> msgTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
>> cacheDesc=null]
>> Registered listeners:
>>
>>
>> Where should we look for the main cause of the problem? What can cause
>> such a behaviour? There seems nothing wrong on server 1 logs etc.
>>
>> We use ignite 1.8.3.
>>
>> --
>> Alper Tekinalp
>>
>> Software Developer
>> Evam Streaming Analytics
>>
>> Atatürk Mah. Turgut Özal Bulv.
>> Gardenya 5 Plaza K:6 Ataşehir
>> 34758 İSTANBUL
>>
>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>> www.evam.com.tr
>> <http://www.evam.com>
>>
>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Best regards,
Andrey V. Mashenkov
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
Can someone point me a direction that why a thread can stuck on the trace
above? Where should I look for? How can I investicate the issue? Where
should I look?
On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi all.
>
>
> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
> connects eachother as server but connects server 1 as client. In a point of
> the time server 3 got stucked at:
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> get0(GridFutureAdapter.java:161)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> get(GridFutureAdapter.java:119)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(
> GridCacheAdapter.java:4665)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(
> GridCacheAdapter.java:1388)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(
> IgniteCacheProxy.java:1121)
> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.evam.cache.client.CachePassthroughInvocationHand
> ler.invokeInternal(CachePassthroughInvocationHandler.java:99)
> at com.evam.cache.client.CachePassthroughInvocationHandler.invoke(
> CachePassthroughInvocationHandler.java:78)
> at com.sun.proxy.$Proxy56.get(Unknown Source)
>
> when getting record from server 1. Long after that server 2 also got
> stucked at the same trace and also server 2 and server 3 disconnects from
> each other.
>
> We investigated gc logs and there is not an unusual behaviour. One things
> is server 1 logs errors as follows when server 2 and server 3 disconnects:
>
> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg -
> Received message without registered handler (will ignore)
> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
> [arr= true, ctx=false, start=0], partId=199, flags=1,
> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0, createTtl=0,
> accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a, locTopVer=AffinityTopologyVersion
> [topVer=33, minorTopVer=2937], msgTopVer=AffinityTopologyVersion
> [topVer=33, minorTopVer=455], cacheDesc=null]
> Registered listeners:
>
>
> Where should we look for the main cause of the problem? What can cause
> such a behaviour? There seems nothing wrong on server 1 logs etc.
>
> We use ignite 1.8.3.
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
Re: Client got stucked on get operation
Posted by Alper Tekinalp <al...@evam.com>.
Hi.
Can someone point me a direction that why a thread can stuck on the trace
above? Where should I look for? How can I investicate the issue? Where
should I look?
On Mon, Mar 20, 2017 at 12:57 PM, Alper Tekinalp <al...@evam.com> wrote:
> Hi all.
>
>
> We have 3 ignite servers. Server 1 works as standalone. Server 2 and 3
> connects eachother as server but connects server 1 as client. In a point of
> the time server 3 got stucked at:
>
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> get0(GridFutureAdapter.java:161)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.
> get(GridFutureAdapter.java:119)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.atomic.GridDhtAtomicCache.get0(GridDhtAtomicCache.java:488)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(
> GridCacheAdapter.java:4665)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(
> GridCacheAdapter.java:1388)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(
> IgniteCacheProxy.java:1121)
> at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.evam.cache.client.CachePassthroughInvocationHand
> ler.invokeInternal(CachePassthroughInvocationHandler.java:99)
> at com.evam.cache.client.CachePassthroughInvocationHandler.invoke(
> CachePassthroughInvocationHandler.java:78)
> at com.sun.proxy.$Proxy56.get(Unknown Source)
>
> when getting record from server 1. Long after that server 2 also got
> stucked at the same trace and also server 2 and server 3 disconnects from
> each other.
>
> We investigated gc logs and there is not an unusual behaviour. One things
> is server 1 logs errors as follows when server 2 and server 3 disconnects:
>
> [ERROR] 2017-03-18 22:01:21.881 [sys-stripe-82-#83%cache-server%] msg -
> Received message without registered handler (will ignore)
> [msg=GridNearSingleGetRequest [futId=1490866022968, key=BinaryObjectImpl
> [arr= true, ctx=false, start=0], partId=199, flags=1,
> topVer=AffinityTopologyVersion [topVer=33, minorTopVer=455],
> subjId=53293ebb-f01b-40b6-a060-bec4209e9c8a, taskNameHash=0, createTtl=0,
> accessTtl=-1], node=53293ebb-f01b-40b6-a060-bec4209e9c8a, locTopVer=AffinityTopologyVersion
> [topVer=33, minorTopVer=2937], msgTopVer=AffinityTopologyVersion
> [topVer=33, minorTopVer=455], cacheDesc=null]
> Registered listeners:
>
>
> Where should we look for the main cause of the problem? What can cause
> such a behaviour? There seems nothing wrong on server 1 logs etc.
>
> We use ignite 1.8.3.
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>