You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Kamlesh Joshi <Ka...@ril.com> on 2020/10/18 06:35:06 UTC
Ignite Cluster got frozen/unreponsive
Hi Igniters,
Our PROD cluster was frozen due to deadlock as per below logs. Couldn't find further info on this, whether is it due to putall/removeall ? for which cache and keys or is it due to some other issue ?
We are using putall and removeall but not doing any ordering while using it.
Could you please check logs, tell us if you find any reason for this deadlock. If it is due to putall/removeall, how to get info for which cache and keys. Please find cluster configuration below,
######IgniteConfiguration###################
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="${IGNITE_INSTANCE_NAME}" />
<property name="peerClassLoadingEnabled" value="true"/>
<property name="metricsLogFrequency" value="0" />
<property name="failureDetectionTimeout" value="60000" />
<property name="clientFailureDetectionTimeout" value="240000" />
<property name="networkTimeout" value="60000" />
############### Cache Configuration ################
<property name="name" value="DeviceDetailsCache" />
<property name="cacheMode" value="PARTITIONED" />
<property name="atomicityMode" value="ATOMIC" />
<property name="backups" value="1" />
<property name="statisticsEnabled" value="true"/>
Below are logs,
[2020-10-17T16:12:17,158][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=53afef47-ff6e-494d-a448-e35cc45b4eff, startPtr=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], checkpointLockWait=0ms, checkpointLockHoldTime=33ms, walCpRecordFsyncDuration=5ms, pages=76968, reason='timeout']
[2020-10-17T16:12:17,224][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000006.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928026.wal]
[2020-10-17T16:12:18,318][INFO ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=53afef47-ff6e-494d-a448-e35cc45b4eff, pages=76968, markPos=FileWALPointer [idx=1928027, fileOff=3575743, len=63453], walSegmentsCleared=5, walSegmentsCovered=[1928022 - 1928026], markDuration=67ms, pagesWrite=706ms, fsync=454ms, total=1227ms]
[2020-10-17T16:12:48,251][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928027, segIdx=7, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]
[2020-10-17T16:12:48,355][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]
[2020-10-17T16:13:27,146][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928028, segIdx=8, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]
[2020-10-17T16:13:27,265][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]
[2020-10-17T16:14:06,135][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928029, segIdx=9, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]
[2020-10-17T16:14:06,250][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]
[2020-10-17T16:15:07,128][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Starting to copy WAL segment [absIdx=1928030, segIdx=0, origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]
[2020-10-17T16:15:07,228][INFO ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager] Copied file [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal, dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]
[2020-10-17T16:15:24,264][WARN ][grid-timeout-worker-#263%EDIFCustomerCC%][G] >>> Possible starvation in striped pool.
Thread name: sys-stripe-40-#41%EDIFCustomerCC%
Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009083, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628085, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=608dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=808dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655252, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227431, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[KeyCacheObjectImpl [part=40, val=null, hasValBytes=true]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cb782ce0571-870075cf-b885-40be-a586-b862991e3798, miniId=db782ce0571-870075cf-b885-40be-a586-b862991e3798, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=fb38f867-4fb3-448b-99db-1f20ca492598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252831927, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821044, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142131082,142131083]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=398dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=598dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628128, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442697, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824023, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227447, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d0c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=01c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628143, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=019dea33571-1d1d3064-dfa1-4587-9131-439530617f71, miniId=119dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250927127, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=d04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, miniId=e04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=68fac742-6ea7-4bb3-9d27-6c626cd4a009, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889898269, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125222995, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9afeae4c-ae1f-47a5-b6e6-032536db894e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250632521, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=41a03e2c-88fa-4ab1-9555-4774118b82a5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901232835, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124902656, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8fff54b3-f00a-4e4e-8421-b81c23b7919e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602137511901, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b00afce2-12e7-41b1-9147-6874d7bdca3e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252085756, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a3e1fb1-37cc-4c16-ace5-34f9d75e8034, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442733, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252434062, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a9252c27-1c4d-4f64-90cb-5b59a7f03eb9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=14061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, miniId=24061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=908a1300-c06b-4349-a94a-e2e4ca836cb0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967622, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=333c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, miniId=433c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1e50ef76-cb94-4ffd-864d-29103600d567, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251239976, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ac6bc567-29d3-40c2-8f62-b63f3e8edba6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901308734, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250879685, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442873, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628186, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824253, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233022, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=34c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, miniId=64c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250414898, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=cab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, miniId=dab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821280, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251065338, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=11395929-053b-4a13-b03f-c8fc670eb8fa, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123820250, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=df729a44-6fd8-4dd1-b9bf-98d0aef3de75, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901376631, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=a1377485-85fd-43ca-96b1-7b323e124662, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251643645, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9cf24c88-db30-43c2-a8ed-2cf14c36f558, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751037, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123619208, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=beddfcbd-19e9-4a74-b87b-153ec35af598, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751050, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821399, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287062, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821413, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821422, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901315682, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0845ba78-0c30-4f97-952d-4c5b57ce8913, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655371, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901303189, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7251566e-d022-4c5e-af08-60feaa3fb9ee, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252832012, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251010602, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967763, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=276f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=676f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2, arr=[142346521,142346523]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821584, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252443040, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900084066, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2e8b3c45-2790-471a-9aa4-2afcdf1f78b5, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602136587142, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c2c02269-cec2-4f96-acdd-2a131a4b27fd, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251177802, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899966390, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6c04a7b6-8b10-4329-b751-92912eaea781, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251968094, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123464026, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044341, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250880196, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233568, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=b53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, miniId=c53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=22c72b15-5e2a-47eb-9949-1d5c7565ddfc, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=7d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, miniId=8d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=e2dff823-0090-40bf-8aab-0a3b5bcf3611, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044373, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=27337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, miniId=57337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=69e1b4e3-5082-4317-8ce0-19c956be89c0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252009770, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250415308, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287470, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=3cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, miniId=5cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95c546c4-3c34-433e-97a5-a24911be098a, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=4f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, miniId=5f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899937408, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f3c978a8-b2fd-443e-b1c0-9197c50521e3, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1, arr=[142346525]]]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287655, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=9e2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, miniId=be2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=4e7366a0-6ba8-4fad-afae-b412601c99b1, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901309460, key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=0d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, miniId=3d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=ce6f4bab-f303-4d87-80a6-c03343400c8e, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124011789, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0b85712f-70b0-4d18-ae92-cf7955ef4464, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252005183, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=5a368e87-d5b1-4aa5-8fdd-cf2184f0a768, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]] Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250950411, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251034391, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252175793, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=3b156c84-9eba-4de0-9861-e4974e17fe1b, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124988181, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=b33c781b-e83a-495c-9e72-5f9d4507a717, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251529166, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=9a3e3502-fa33-494b-a445-b6ec254e5d59, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124901688, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=31f69255-0587-49b8-bf74-8d9941967eb4, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251847453, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889551081, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251845149, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123481727, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest [futId=1085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, miniId=3085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=1769deb3-fd7b-4b36-9707-1808245f6fe9, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251448830, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=44b5b67c-da49-468b-96af-e302b07b6a16, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251201103, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250596135, key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=653fbfe7-dd96-4896-9b63-9b5fd3aec77f, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602125326244, key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0], subjId=c55ebb6d-4539-44e2-b14f-24a9bb10f077, taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]]]
Deadlock: true
Completed: 58842256
Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]
Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2930)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1746)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3138)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:135)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)
at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=sys-stripe-120, blockedFor=77s]
[2020-10-17T16:15:51,276][WARN ][disco-event-worker-#342%EDIFCustomerCC%][G] Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING, blockCnt=7120, waitCnt=58416728]
Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8, ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]
[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]]]
org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false, heartbeatTs=1602931473886]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) [ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) [ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) [ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) [ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2827) [ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.6.jar:2.7.6]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2020-10-17T16:15:51,278][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Deadlocked threads detected (see thread dump below) [deadlockedThreadsCnt=2]
[2020-10-17T16:15:55,716][WARN ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerCC] Possible too long JVM pause: 4429 milliseconds.
[2020-10-17T16:15:55,760][WARN ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Thread dump at 2020/10/17 16:15:55 IST
Thread [name="sys-#194498%EDIFCustomerCC%", id=382762, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-#194497%EDIFCustomerCC%", id=382761, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks and Regards,
Kamlesh Joshi
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Re: Ignite Cluster got frozen/unreponsive
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This looks like a transactional deadlock. Is this operation a part of
transaction? Are you locking keys in the same order?
Regards,
--
Ilya Kasnacheev
вс, 18 окт. 2020 г. в 09:35, Kamlesh Joshi <Ka...@ril.com>:
> Hi Igniters,
>
>
>
> Our PROD cluster was frozen due to deadlock as per below logs. Couldn't
> find further info on this, whether is it due to putall/removeall ? for
> which cache and keys or is it due to some other issue ?
>
> We are using putall and removeall but not doing any ordering while using
> it.
>
>
>
> Could you please check logs, tell us if you find any reason for this
> deadlock. If it is due to putall/removeall, how to get info for which cache
> and keys. Please find cluster configuration below,
>
>
>
> *######IgniteConfiguration###################*
>
> *<bean class="org.apache.ignite.configuration.IgniteConfiguration">*
>
> * <property name="igniteInstanceName"
> value="${IGNITE_INSTANCE_NAME}" />*
>
> * <property name="peerClassLoadingEnabled" value="true"/>*
>
> * <property name="metricsLogFrequency" value="0" />*
>
> * <property name="failureDetectionTimeout" value="60000" />*
>
> * <property name="clientFailureDetectionTimeout" value="240000" />*
>
> * <property name="networkTimeout" value="60000" />*
>
>
>
>
>
> *############### Cache Configuration ################*
>
> *<property name="name" value="DeviceDetailsCache" />*
>
> * <property name="cacheMode" value="PARTITIONED" />*
>
> * <property name="atomicityMode" value="ATOMIC" />*
>
> * <property name="backups" value="1" />*
>
> * <property name="statisticsEnabled" value="true"/>*
>
>
>
> *Below are logs,*
>
>
>
> *[2020-10-17T16:12:17,158][INFO
> ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager]
> Checkpoint started [checkpointId=53afef47-ff6e-494d-a448-e35cc45b4eff,
> startPtr=FileWALPointer [idx=1928027, fileOff=3575743, len=63453],
> checkpointLockWait=0ms, checkpointLockHoldTime=33ms,
> walCpRecordFsyncDuration=5ms, pages=76968, reason='timeout']*
>
> *[2020-10-17T16:12:17,224][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Copied file
> [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000006.wal,
> dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928026.wal]*
>
> *[2020-10-17T16:12:18,318][INFO
> ][db-checkpoint-thread-#385%EDIFCustomerCC%][GridCacheDatabaseSharedManager]
> Checkpoint finished [cpId=53afef47-ff6e-494d-a448-e35cc45b4eff,
> pages=76968, markPos=FileWALPointer [idx=1928027, fileOff=3575743,
> len=63453], walSegmentsCleared=5, walSegmentsCovered=[1928022 - 1928026],
> markDuration=67ms, pagesWrite=706ms, fsync=454ms, total=1227ms]*
>
> *[2020-10-17T16:12:48,251][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Starting to copy WAL segment [absIdx=1928027, segIdx=7,
> origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal,
> dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]*
>
> *[2020-10-17T16:12:48,355][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Copied file
> [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000007.wal,
> dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928027.wal]*
>
> *[2020-10-17T16:13:27,146][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Starting to copy WAL segment [absIdx=1928028, segIdx=8,
> origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal,
> dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]*
>
> *[2020-10-17T16:13:27,265][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Copied file
> [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000008.wal,
> dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928028.wal]*
>
> *[2020-10-17T16:14:06,135][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Starting to copy WAL segment [absIdx=1928029, segIdx=9,
> origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal,
> dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]*
>
> *[2020-10-17T16:14:06,250][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Copied file
> [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000009.wal,
> dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928029.wal]*
>
> *[2020-10-17T16:15:07,128][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Starting to copy WAL segment [absIdx=1928030, segIdx=0,
> origFile=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal,
> dstFile=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]*
>
> *[2020-10-17T16:15:07,228][INFO
> ][wal-file-archiver%EDIFCustomerCC-#360%EDIFCustomerCC%][FileWriteAheadLogManager]
> Copied file
> [src=/datastore1/wal/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000000000000.wal,
> dst=/datastore1/archive/node00-34dda35b-606f-4124-936e-7b1a01531043/0000000001928030.wal]*
>
> *[2020-10-17T16:15:24,264][WARN
> ][grid-timeout-worker-#263%EDIFCustomerCC%][G] >>> Possible starvation in
> striped pool.*
>
> * Thread name: sys-stripe-40-#41%EDIFCustomerCC%*
>
> * Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridNearSingleGetRequest [futId=1602252009083, key=KeyCacheObjectImpl
> [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628085,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=608dea33571-1d1d3064-dfa1-4587-9131-439530617f71,
> miniId=808dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251655252, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227431,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[KeyCacheObjectImpl [part=40, val=null, hasValBytes=true]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=cb782ce0571-870075cf-b885-40be-a586-b862991e3798,
> miniId=db782ce0571-870075cf-b885-40be-a586-b862991e3798, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=fb38f867-4fb3-448b-99db-1f20ca492598,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602252831927, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0], BinaryObjectImpl
> [arr= true, ctx=false, start=0]], conflictTtls=null,
> conflictExpireTimes=null, expiryPlc=null, initSize=0, filter=null,
> parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]],
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251821044, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true,
> ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null,
> expiryPlc=null, initSize=0, filter=null,
> parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]],
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true,
> ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null,
> expiryPlc=null, initSize=0, filter=null,
> parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]],
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2,
> arr=[142131082,142131083]]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=398dea33571-1d1d3064-dfa1-4587-9131-439530617f71,
> miniId=598dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602250628128, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442697,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824023,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901227447,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6b2fb754-9165-4d61-891f-5970e263c396, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=d0c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574,
> miniId=01c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602250628143, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=3495693d-2021-4b80-9322-97227ab02d7a,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true,
> ctx=false, start=0]], conflictTtls=null, conflictExpireTimes=null,
> expiryPlc=null, initSize=0, filter=null,
> parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=keepBinary]]]],
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=019dea33571-1d1d3064-dfa1-4587-9131-439530617f71,
> miniId=119dea33571-1d1d3064-dfa1-4587-9131-439530617f71, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602250927127, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=d04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8,
> miniId=e04e1e33571-1f7c74a2-aff6-434b-a347-d88f70581bb8, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=68fac742-6ea7-4bb3-9d27-6c626cd4a009,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602889898269, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=a745cdf1-ff3f-4e0f-9c97-d46319342227,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602125222995, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=9afeae4c-ae1f-47a5-b6e6-032536db894e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250632521,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=41a03e2c-88fa-4ab1-9555-4774118b82a5, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901232835,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124902656,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=8fff54b3-f00a-4e4e-8421-b81c23b7919e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602137511901,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=b00afce2-12e7-41b1-9147-6874d7bdca3e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252085756,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=2a3e1fb1-37cc-4c16-ace5-34f9d75e8034, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442733,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252434062,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=a9252c27-1c4d-4f64-90cb-5b59a7f03eb9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=14061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03,
> miniId=24061b33571-81fe5115-bee4-4ff8-8dc0-3b38d25cca03, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=908a1300-c06b-4349-a94a-e2e4ca836cb0,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251967622, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=921cd171-9236-4257-9bf0-81ffd57d8555,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=333c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644,
> miniId=433c0b33571-bfe09cc5-6435-41be-a319-d4e6f5999644, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=1e50ef76-cb94-4ffd-864d-29103600d567,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251239976, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=ac6bc567-29d3-40c2-8f62-b63f3e8edba6,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602901308734, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250879685,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252442873,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250628186,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=3495693d-2021-4b80-9322-97227ab02d7a, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearAtomicFullUpdateRequest
> [keys=[BinaryObjectImpl [arr= true, ctx=false, start=0]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251824253,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233022,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=34c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574,
> miniId=64c22e33571-e189a141-f952-4b63-a8ed-395f16b0a574, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=1ac4a008-e359-487e-bd60-8bc4d101085c,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602250414898, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=2a66a450-c90c-44e5-826b-ab0be7706e88,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=cab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd,
> miniId=dab59a33571-7fc24c46-bc64-4e80-a4fc-6d6f02d8d8cd, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251821280, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251065338, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=11395929-053b-4a13-b03f-c8fc670eb8fa, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123820250,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=df729a44-6fd8-4dd1-b9bf-98d0aef3de75, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901376631,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=a1377485-85fd-43ca-96b1-7b323e124662, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251643645,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=9cf24c88-db30-43c2-a8ed-2cf14c36f558, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751037,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123619208,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=beddfcbd-19e9-4a74-b87b-153ec35af598, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123751050,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=d2d8e1bf-43a9-47b0-bc91-97a9fb0fa498, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821399,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287062,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821413,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821422,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901315682,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=0845ba78-0c30-4f97-952d-4c5b57ce8913, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251655371,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=d31dfbf3-95a4-42c8-be02-99cc24d174bd, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901303189,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=7251566e-d022-4c5e-af08-60feaa3fb9ee, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252832012,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=e5667fd1-0d1d-466a-aafd-3d4daafccb6b, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251010602,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251967763,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=276f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8,
> miniId=676f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=2,
> arr=[142346521,142346523]]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251821584,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252443040,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=cca2a7c6-0e27-40d3-8782-3b964b41629e, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900084066,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=2e8b3c45-2790-471a-9aa4-2afcdf1f78b5, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602136587142,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=c2c02269-cec2-4f96-acdd-2a131a4b27fd, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251177802,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899966390,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6c04a7b6-8b10-4329-b751-92912eaea781, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251968094,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=921cd171-9236-4257-9bf0-81ffd57d8555, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123464026,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602900044341,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250880196,
> key=KeyCacheObjectImpl [part=40, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=7da2bd3b-fb71-4d3f-a634-8fa9c6d60d14, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901233568,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=e01970ae-87af-4602-b303-7a75c2e64aa7, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=b53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6,
> miniId=c53d3e33571-701b81e6-a49b-44df-81c4-a063a7d1e7a6, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=22c72b15-5e2a-47eb-9949-1d5c7565ddfc,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=7d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4,
> miniId=8d4a6243571-6fff9fa2-9496-41bd-bf6e-db2d52c4a2e4, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=e2dff823-0090-40bf-8aab-0a3b5bcf3611,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602900044373, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=8787e0aa-ae20-4c7d-bb5b-c8be99c88a3d, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=27337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc,
> miniId=57337243571-cdc9b3fd-927b-4c38-b7ac-52840c0d42cc, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=69e1b4e3-5082-4317-8ce0-19c956be89c0,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602252009770, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=236ba74c-3e77-43db-b06d-ae757784e7d6, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250415308,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=2a66a450-c90c-44e5-826b-ab0be7706e88, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602901287470,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=3cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed,
> miniId=5cdf3e33571-8b9a9b5e-83cf-469d-b424-4feba04aa2ed, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=95c546c4-3c34-433e-97a5-a24911be098a,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=4f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8,
> miniId=5f7f3e33571-3bd539ba-2f6c-41d9-ba62-80ec8d57c2f8, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=425f44e2-2600-427b-a47a-1823af73692d,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridNearAtomicFullUpdateRequest [keys=[BinaryObjectImpl [arr= true,
> ctx=false, start=0], BinaryObjectImpl [arr= true, ctx=false, start=0]],
> conflictTtls=null, conflictExpireTimes=null, expiryPlc=null, initSize=0,
> filter=null, parent=GridNearAtomicAbstractUpdateRequest [res=null,
> flags=keepBinary]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602899937408,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=f3c978a8-b2fd-443e-b1c0-9197c50521e3, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse
> [futIds=GridLongList [idx=1, arr=[142346525]]]]], Message closure
> [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
> timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602901287655, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=b5e6888c-3f7b-4cec-95e0-5b1acf2da830,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=9e2f3e33571-055b3332-1a13-40cf-8857-662219beebf6,
> miniId=be2f3e33571-055b3332-1a13-40cf-8857-662219beebf6, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=4e7366a0-6ba8-4fad-afae-b412601c99b1,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602901309460, key=KeyCacheObjectImpl [part=40, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=6bbfc872-5022-48dc-b4a6-f8dabec35cf0,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearGetRequest
> [futId=0d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82,
> miniId=3d2f1e33571-b536da48-556d-4c6b-be8f-817b8280fb82, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=ce6f4bab-f303-4d87-80a6-c03343400c8e,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602124011789, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=0b85712f-70b0-4d18-ae92-cf7955ef4464, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252005183,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=5a368e87-d5b1-4aa5-8fdd-cf2184f0a768, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]] Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602250950411,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=0498547b-c945-4c65-908a-6ac0ddf4dff8, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251034391,
> key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=f94e3dca-bf5c-44df-bb18-87fc3a9a8ca0, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602252175793,
> key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=3b156c84-9eba-4de0-9861-e4974e17fe1b, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124988181,
> key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=b33c781b-e83a-495c-9e72-5f9d4507a717, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251529166,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=9a3e3502-fa33-494b-a445-b6ec254e5d59, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602124901688,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=31f69255-0587-49b8-bf74-8d9941967eb4, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251847453,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=6a3462e2-b41f-47e4-b34c-db11ce061138, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602889551081,
> key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=bff99e14-d0a6-488b-9e5b-7ab3b2190382, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602251845149,
> key=BinaryObjectImpl [arr= true, ctx=false, start=0], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=376c2540-8cd8-4cfc-8a6d-6e71f22fd3c9, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1602123481727,
> key=KeyCacheObjectImpl [part=120, val=null, hasValBytes=true], flags=1,
> topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=95dc0be1-d639-4920-a635-3b28075ce7e7, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]], Message closure [msg=GridIoMessage
> [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridNearGetRequest
> [futId=1085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7,
> miniId=3085ba33571-e48b4a3d-7144-4e73-a0c6-f2c452b56cd7, ver=null,
> keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=1769deb3-fd7b-4b36-9707-1808245f6fe9,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251448830, key=KeyCacheObjectImpl [part=120, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=44b5b67c-da49-468b-96af-e302b07b6a16,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602251201103, key=KeyCacheObjectImpl [part=120, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=db5b0c2d-48b5-444f-a59b-7850a2b6073c,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602250596135, key=KeyCacheObjectImpl [part=120, val=null,
> hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=956,
> minorTopVer=0], subjId=653fbfe7-dd96-4896-9b63-9b5fd3aec77f,
> taskNameHash=0, createTtl=-1, accessTtl=-1, mvccSnapshot=null]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1602125326244, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=956, minorTopVer=0],
> subjId=c55ebb6d-4539-44e2-b14f-24a9bb10f077, taskNameHash=0, createTtl=-1,
> accessTtl=-1, mvccSnapshot=null]]]]*
>
> * Deadlock: true*
>
> * Completed: 58842256*
>
> *Thread [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153,
> state=WAITING, blockCnt=7120, waitCnt=58416728]*
>
> * Lock
> [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8,
> ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]*
>
> * at sun.misc.Unsafe.park(Native Method)*
>
> * at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)*
>
> * at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)*
>
> * at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)*
>
> * at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)*
>
> * at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)*
>
> * at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)*
>
> * at
> o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:4967)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2930)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1746)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3138)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:135)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271)*
>
> * at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)*
>
> * at
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)*
>
> * at
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)*
>
> * at
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)*
>
> * at
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)*
>
> * at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)*
>
> * at
> o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)*
>
> * at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)*
>
> * at java.lang.Thread.run(Thread.java:748)*
>
>
>
> *[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-120,
> blockedFor=77s]*
>
> *[2020-10-17T16:15:51,276][WARN
> ][disco-event-worker-#342%EDIFCustomerCC%][G] Thread
> [name="sys-stripe-120-#121%EDIFCustomerCC%", id=153, state=WAITING,
> blockCnt=7120, waitCnt=58416728]*
>
> * Lock
> [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@793496f8,
> ownerName=sys-stripe-40-#41%EDIFCustomerCC%, ownerId=73]*
>
>
>
> *[2020-10-17T16:15:51,276][ERROR][disco-event-worker-#342%EDIFCustomerCC%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-120, igniteInstanceName=EDIFCustomerCC, finished=false,
> heartbeatTs=1602931473886]]]*
>
> *org.apache.ignite.IgniteException: GridWorker [name=sys-stripe-120,
> igniteInstanceName=EDIFCustomerCC, finished=false,
> heartbeatTs=1602931473886]*
>
> * at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2827)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [ignite-core-2.7.6.jar:2.7.6]*
>
> * at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]*
>
> *[2020-10-17T16:15:51,278][WARN
> ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Deadlocked
> threads detected (see thread dump below) [deadlockedThreadsCnt=2]*
>
> *[2020-10-17T16:15:55,716][WARN
> ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomerCC] Possible too long
> JVM pause: 4429 milliseconds.*
>
> *[2020-10-17T16:15:55,760][WARN
> ][disco-event-worker-#342%EDIFCustomerCC%][FailureProcessor] Thread dump at
> 2020/10/17 16:15:55 IST*
>
> *Thread [name="sys-#194498%EDIFCustomerCC%", id=382762,
> state=TIMED_WAITING, blockCnt=0, waitCnt=1]*
>
> * Lock
> [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3,
> ownerName=null, ownerId=-1]*
>
> * at sun.misc.Unsafe.park(Native Method)*
>
> * at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)*
>
> * at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)*
>
> * at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
>
> * at java.lang.Thread.run(Thread.java:748)*
>
>
>
> *Thread [name="sys-#194497%EDIFCustomerCC%", id=382761,
> state=TIMED_WAITING, blockCnt=0, waitCnt=1]*
>
> * Lock
> [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@435ed2a3,
> ownerName=null, ownerId=-1]*
>
> * at sun.misc.Unsafe.park(Native Method)*
>
> * at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)*
>
> * at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)*
>
> * at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)*
>
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
>
> * at java.lang.Thread.run(Thread.java:748)*
>
>
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>