You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ksenia Rybakova (JIRA)" <ji...@apache.org> on 2016/11/30 12:38:59 UTC
[jira] [Commented] (IGNITE-3269) Benchmark driver stops working
when one of servers left grid and backupcount 0
[ https://issues.apache.org/jira/browse/IGNITE-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708483#comment-15708483 ]
Ksenia Rybakova commented on IGNITE-3269:
-----------------------------------------
Similar case:
ignite-1.8, 80 clients and 20 servers, backup count=1:
{noformat}
[15:23:36,733][INFO ][grid-timeout-worker-#7%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=215e4b73, name=null, uptime=00:19:16:915]
^-- H/N/C [hosts=8, nodes=100, CPUs=128]
^-- CPU [cur=1.53%, avg=2%, GC=0%]
^-- Heap [used=437MB, free=57.23%, comm=1024MB]
^-- Non heap [used=44MB, free=65.29%, comm=47MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[15:23:37,029][WARN ][grid-nio-worker-1-#10%null%][TcpCommunicationSpi] Failed to process selector key (will close): GridSelectorNioSessionImpl [selectorIdx=1, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], r
eadBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], recovery=GridNioRecoveryDescriptor [acked=46848, resendCnt=0, rcvCnt=46758, sentCnt=46860, reserved=true, lastAck=46752, nodeLeft=false, node=TcpDiscoveryNode [id=d4069148-449b
-41ee-b772-d5586265f23f, addrs=[127.0.0.1, 172.25.1.38], sockAddrs=[/172.25.1.38:47500, /127.0.0.1:47500], discPort=47500, order=102, intOrder=101, lastExchangeTime=1480508020664, loc=false, ver=1.8.0#20161129-sha1:d0e0eaac, isClient=fal
se], connected=true, connectCnt=1, queueLimit=5120, reserveCnt=1], super=GridNioSessionImpl [locAddr=/172.25.1.35:37134, rmtAddr=/172.25.1.38:47100, createTime=1480508073194, closeTime=0, bytesSent=52124554, bytesRcvd=24331807, sndSchedT
ime=1480508617009, lastSndTime=1480508617019, lastRcvTime=1480508616898, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@76ae88f4, directMode=true], GridConnectionBytesVeri
fyFilter], accepted=false]]
[15:23:37,030][WARN ][grid-nio-worker-1-#10%null%][TcpCommunicationSpi] Closing NIO session because of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[15:23:37,089][WARN ][tcp-comm-worker-#1%null%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/172.25.1.38:47100, failureDetectionTimeout=30000]
[15:23:37,090][WARN ][tcp-comm-worker-#1%null%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/172.25.1.38:4
7100, /127.0.0.1:47100]]
[15:23:37] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=3204e25b851-d7870206-d4cd-4c35-a749-d1d9b5b628bc, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
pped=false, needCheckBackup=null, hasRemoteLocks=true, thread=<failed to find active thread 299>, mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=GridLongList [idx=3, arr=[3716,107944136,-535738254]], txMap={IgniteTxKey [key=o.a.i.yardstick.cache.lo
ad.model.key.Identifier [idHash=454392180, hash=-1688540165, id=1663029, code=id 1663029], cacheId=3716]=IgniteTxEntry [key=o.a.i.yardstick.cache.load.model.key.Identifier [idHash=454392180, hash=-1688540165, id=1663029, code=id 1663029]
, cacheId=3716, partId=5, txKey=IgniteTxKey [key=o.a.i.yardstick.cache.load.model.key.Identifier [idHash=454392180, hash=-1688540165, id=1663029, code=id 1663029], cacheId=3716], val=[op=CREATE, val=o.a.i.yardstick.cache.model.Account [i
dHash=517520528, hash=216730851, balance=1663029]], prevVal=[op=CREATE, val=o.a.i.yardstick.cache.model.Account [idHash=517520528, hash=216730851, balance=1663029]], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictEx
pireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=o.a.i.yardstick.cache.load.mod
el.key.Identifier [idHash=454392180, hash=-1688540165, id=1663029, code=id 1663029], val=null, startVer=1480614477546, ver=GridCacheVersion [topVer=91987517, time=1480508616519, order=1480614477371, nodeOrder=1], hash=-1688540165, extras
=null, flags=0]]], prepared=0, locked=true, nodeId=5650c65d-918a-4a27-af67-14d995de536b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time
=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1690740, hasValBytes=false], cacheId=107944136]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1690740, hasValBytes=false], cacheId=107944136, part
Id=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1690740, hasValBytes=false], cacheId=107944136], val=[op=DELETE, val=null], prevVal=[op=DELETE, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireT
ime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1690740, ha
sValBytes=false], val=null, startVer=1480614477560, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477331, nodeOrder=12], hash=1690740, extras=null, flags=0]]], prepared=0, locked=true, nodeId=4b092c35-1577-48f1-
ae3c-7edd4138de60, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=
KeyCacheObjectImpl [val=112205, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=112205, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=112205, hasValBy
tes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], fil
tersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=112205, hasValBytes=false], val=o.a.i.yardstick.cache.model.Account [idHash=6587437
46, hash=1027965626, balance=112205], startVer=1480614477561, ver=GridCacheVersion [topVer=91987517, time=1480508341078, order=1480605489332, nodeOrder=20], hash=1449620069, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94abd5
a8-4557-4d97-bdbf-6240a734b8c9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], Igni
teTxKey [key=KeyCacheObjectImpl [val=1162431, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1162431, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1
162431, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1162431, hasValBytes=false], val=null, startVer=1480614477562, ver=
GridCacheVersion [topVer=91987517, time=1480508616564, order=1480614480087, nodeOrder=20], hash=1992247094, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94abd5a8-4557-4d97-bdbf-6240a734b8c9, locMapped=false, expiryPlc=null, t
ransferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1174231, hasValBytes=false],
cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1174231, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1174231, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, v
al=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDe
tachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1174231, hasValBytes=false], val=null, startVer=1480614477563, ver=GridCacheVersion [topVer=91987517, time=1480508616568, order=14806
14480101, nodeOrder=5], hash=1993228275, extras=null, flags=0]]], prepared=0, locked=true, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=118536, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=11853
6, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=118536, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], ent
ryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapE
ntry [key=KeyCacheObjectImpl [val=118536, hasValBytes=false], val=null, startVer=1480614477564, ver=GridCacheVersion [topVer=91987517, time=1480508608044, order=1480614212713, nodeOrder=11], hash=1449801792, extras=null, flags=0]]], prep
ared=0, locked=true, nodeId=4f7e0f8b-a895-4971-84e3-bcb80d3fbc24, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1
480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1207848, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1207848, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTx
Key [key=KeyCacheObjectImpl [val=1207848, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1207848, hasValBytes=false], val=null, startVer=1480614477565, ver=GridCacheVersion [topVer=91987517, time=1480508616619, order=1480614483083, nodeOrder=16], hash=2015487956, extras=null, flags=0]]], prepared=0, locked=true, nodeId=143f09ca-6fab-4c85-aaff-5f023a17ece2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1242788, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1242788, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1242788, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1242788, hasValBytes=false], val=null, startVer=1480614477566, ver=GridCacheVersion [topVer=91987517, time=1480508616615, order=1480614483239, nodeOrder=18], hash=2019032248, extras=null, flags=0]]], prepared=0, locked=true, nodeId=7e96fb15-50ac-47f7-a0cf-6827b808f744, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1265966, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1265966, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1265966, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1265966, hasValBytes=false], val=null, startVer=1480614477567, ver=GridCacheVersion [topVer=91987517, time=1480508616626, order=1480614483416, nodeOrder=6], hash=2020970521, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94619cb7-cb3f-4a60-bc22-2681f933c37a, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1284801, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1284801, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1284801, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1284801, hasValBytes=false], val=null, startVer=1480614477568, ver=GridCacheVersion [topVer=91987517, time=1480508616635, order=1480614483598, nodeOrder=4], hash=2022786620, extras=null, flags=0]]], prepared=0, locked=true, nodeId=a2ef8fdd-35e4-4731-805e-f0292dba9085, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1317631, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1317631, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1317631, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1317631, hasValBytes=false], val=null, startVer=1480614477569, ver=GridCacheVersion [topVer=91987517, time=1480508616649, order=1480614484494, nodeOrder=2], hash=2045038668, extras=null, flags=0]]], prepared=0, locked=true, nodeId=a394b7ab-d6b3-4cd8-9b4f-8efef17957a0, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1432033, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1432033, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1432033, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1432033, hasValBytes=false], val=null, startVer=1480614477570, ver=GridCacheVersion [topVer=91987517, time=1480508616691, order=1480614485503, nodeOrder=7], hash=2075360142, extras=null, flags=0]]], prepared=0, locked=true, nodeId=26199f94-1405-402e-a52e-c8c232d4dc93, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1464612, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1464612, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1464612, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1464612, hasValBytes=false], val=null, startVer=1480614477571, ver=GridCacheVersion [topVer=91987517, time=1480508616704, order=1480614486144, nodeOrder=2], hash=2078195990, extras=null, flags=0]]], prepared=0, locked=true, nodeId=a394b7ab-d6b3-4cd8-9b4f-8efef17957a0, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1505597, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1505597, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1505597, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1505597, hasValBytes=false], val=null, startVer=1480614477572, ver=GridCacheVersion [topVer=91987517, time=1480508616746, order=1480614487216, nodeOrder=13], hash=2101313098, extras=null, flags=0]]], prepared=0, locked=true, nodeId=89f23db4-4835-479a-b552-4f3b7a2a1264, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1509945, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1509945, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1509945, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1509945, hasValBytes=false], val=null, startVer=1480614477573, ver=GridCacheVersion [topVer=91987517, time=1480508616750, order=1480614487286, nodeOrder=9], hash=2101435949, extras=null, flags=0]]], prepared=0, locked=true, nodeId=adc11bfc-5326-433a-9430-c6bc32c679f9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1525290, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1525290, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1525290, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1525290, hasValBytes=false], val=null, startVer=1480614477574, ver=GridCacheVersion [topVer=91987517, time=1480508616752, order=1480614487368, nodeOrder=5], hash=2103157250, extras=null, flags=0]]], prepared=0, locked=true, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1532049, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1532049, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1532049, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1532049, hasValBytes=false], val=null, startVer=1480614477575, ver=GridCacheVersion [topVer=91987517, time=1480508616752, order=1480614487432, nodeOrder=9], hash=2103989330, extras=null, flags=0]]], prepared=0, locked=true, nodeId=adc11bfc-5326-433a-9430-c6bc32c679f9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1535605, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1535605, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1535605, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1535605, hasValBytes=false], val=null, startVer=1480614477576, ver=GridCacheVersion [topVer=91987517, time=1480508616753, order=1480614487451, nodeOrder=7], hash=2104084341, extras=null, flags=0]]], prepared=0, locked=true, nodeId=26199f94-1405-402e-a52e-c8c232d4dc93, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1545573, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1545573, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1545573, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1545573, hasValBytes=false], val=null, startVer=1480614477577, ver=GridCacheVersion [topVer=91987517, time=1480508616754, order=1480614487465, nodeOrder=5], hash=2105007116, extras=null, flags=0]]], prepared=0, locked=true, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1550926, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1550926, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1550926, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1550926, hasValBytes=false], val=null, startVer=1480614477578, ver=GridCacheVersion [topVer=91987517, time=1480508616758, order=1480614487471, nodeOrder=1], hash=2105785374, extras=null, flags=0]]], prepared=0, locked=true, nodeId=5650c65d-918a-4a27-af67-14d995de536b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1567455, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1567455, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1567455, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1567455, hasValBytes=false], val=null, startVer=1480614477579, ver=GridCacheVersion [topVer=91987517, time=1480508616760, order=1480614487790, nodeOrder=20], hash=2106912719, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94abd5a8-4557-4d97-bdbf-6240a734b8c9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1591928, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1591928, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1591928, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1591928, hasValBytes=false], val=null, startVer=1480614477580, ver=GridCacheVersion [topVer=91987517, time=1480508616810, order=1480614489088, nodeOrder=19], hash=2109509251, extras=null, flags=0]]], prepared=0, locked=true, nodeId=017cbadc-cc88-4995-b008-18482f75025c, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1656353, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1656353, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1656353, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1656353, hasValBytes=false], val=null, startVer=1480614477581, ver=GridCacheVersion [topVer=91987517, time=1480508616935, order=1480614492609, nodeOrder=3], hash=2134587595, extras=null, flags=0]]], prepared=0, locked=true, nodeId=b3e4f19e-a359-4649-8040-f0139e0ba1bc, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1703739, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1703739, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1703739, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1703739, hasValBytes=false], val=null, startVer=1480614477582, ver=GridCacheVersion [topVer=91987517, time=1480508616945, order=1480614492877, nodeOrder=15], hash=-2136453740, extras=null, flags=0]]], prepared=0, locked=true, nodeId=5640dff9-f585-45e3-bfc5-850365f74bac, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1706847, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1706847, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1706847, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1706847, hasValBytes=false], val=null, startVer=1480614477583, ver=GridCacheVersion [topVer=91987517, time=1480508616945, order=1480614492895, nodeOrder=9], hash=-2136363377, extras=null, flags=0]]], prepared=0, locked=true, nodeId=adc11bfc-5326-433a-9430-c6bc32c679f9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=171015, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=171015, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=171015, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=171015, hasValBytes=false], val=o.a.i.yardstick.cache.model.Person1 [idHash=945522298, hash=171015, val1=171015], startVer=1480614477584, ver=GridCacheVersion [topVer=91987416, time=0, order=0, nodeOrder=1], hash=1455129513, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94619cb7-cb3f-4a60-bc22-2681f933c37a, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1735928, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1735928, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1735928, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1735928, hasValBytes=false], val=null, startVer=1480614477585, ver=GridCacheVersion [topVer=91987517, time=1480508616948, order=1480614492933, nodeOrder=12], hash=-2133621705, extras=null, flags=0]]], prepared=0, locked=true, nodeId=4b092c35-1577-48f1-ae3c-7edd4138de60, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1758443, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1758443, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1758443, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1758443, hasValBytes=false], val=null, startVer=1480614477586, ver=GridCacheVersion [topVer=91987517, time=1480508616948, order=1480614492952, nodeOrder=20], hash=-2131690038, extras=null, flags=0]]], prepared=0, locked=true, nodeId=94abd5a8-4557-4d97-bdbf-6240a734b8c9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=1892566, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=1892566, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=1892566, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=1892566, hasValBytes=false], val=null, startVer=1480614477587, ver=GridCacheVersion [topVer=91987517, time=1480508616951, order=1480614493049, nodeOrder=3], hash=-2099544523, extras=null, flags=0]]], prepared=0, locked=true, nodeId=b3e4f19e-a359-4649-8040-f0139e0ba1bc, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=257975, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=257975, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=257975, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=257975, hasValBytes=false], val=o.a.i.yardstick.cache.model.Account [idHash=14274686, hash=223338103, balance=257975], startVer=1480614477588, ver=GridCacheVersion [topVer=91987416, time=0, order=0, nodeOrder=1], hash=1482099203, extras=null, flags=0]]], prepared=0, locked=true, nodeId=26199f94-1405-402e-a52e-c8c232d4dc93, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=296298, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=296298, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=296298, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=296298, hasValBytes=false], val=null, startVer=1480614477589, ver=GridCacheVersion [topVer=91987517, time=1480508577134, order=1480613223582, nodeOrder=18], hash=1485756834, extras=null, flags=0]]], prepared=0, locked=true, nodeId=7e96fb15-50ac-47f7-a0cf-6827b808f744, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=311640, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=311640, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=311640, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=311640, hasValBytes=false], val=null, startVer=1480614477590, ver=GridCacheVersion [topVer=91987517, time=1480508554555, order=1480612459295, nodeOrder=19], hash=1506852543, extras=null, flags=0]]], prepared=0, locked=true, nodeId=017cbadc-cc88-4995-b008-18482f75025c, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=316145, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=316145, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=316145, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=316145, hasValBytes=false], val=o.a.i.yardstick.cache.model.Account [idHash=1543962785, hash=679715961, balance=316145], startVer=1480614477591, ver=GridCacheVersion [topVer=91987416, time=0, order=0, nodeOrder=1], hash=1506996698, extras=null, flags=0]]], prepared=0, locked=true, nodeId=143f09ca-6fab-4c85-aaff-5f023a17ece2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=356344, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=356344, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=356344, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=356344, hasValBytes=false], val=null, startVer=1480614477592, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477592, nodeOrder=72], hash=1510692703, extras=null, flags=0]]], prepared=0, locked=false, nodeId=d4069148-449b-41ee-b772-d5586265f23f, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=383068, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=383068, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=383068, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=383068, hasValBytes=false], val=null, startVer=1480614477593, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477593, nodeOrder=72], hash=1513371076, extras=null, flags=0]]], prepared=0, locked=false, nodeId=b3e4f19e-a359-4649-8040-f0139e0ba1bc, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=405321, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=405321, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=405321, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=405321, hasValBytes=false], val=null, startVer=1480614477594, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477594, nodeOrder=72], hash=1534674393, extras=null, flags=0]]], prepared=0, locked=false, nodeId=d4069148-449b-41ee-b772-d5586265f23f, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=478926, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=478926, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=478926, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=478926, hasValBytes=false], val=null, startVer=1480614477595, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477595, nodeOrder=72], hash=1541234184, extras=null, flags=0]]], prepared=0, locked=false, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=482088, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=482088, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=482088, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=482088, hasValBytes=false], val=null, startVer=1480614477596, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477596, nodeOrder=72], hash=1541970498, extras=null, flags=0]]], prepared=0, locked=false, nodeId=017cbadc-cc88-4995-b008-18482f75025c, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=510491, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=510491, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=510491, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=510491, hasValBytes=false], val=null, startVer=1480614477597, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477597, nodeOrder=72], hash=1564079288, extras=null, flags=0]]], prepared=0, locked=false, nodeId=7e96fb15-50ac-47f7-a0cf-6827b808f744, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=539569, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=539569, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=539569, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=539569, hasValBytes=false], val=null, startVer=1480614477598, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477598, nodeOrder=72], hash=1566195325, extras=null, flags=0]]], prepared=0, locked=false, nodeId=94619cb7-cb3f-4a60-bc22-2681f933c37a, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=547871, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=547871, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=547871, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=547871, hasValBytes=false], val=null, startVer=1480614477599, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477599, nodeOrder=72], hash=1567062170, extras=null, flags=0]]], prepared=0, locked=false, nodeId=5650c65d-918a-4a27-af67-14d995de536b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=572222, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=572222, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=572222, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=572222, hasValBytes=false], val=null, startVer=1480614477600, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477600, nodeOrder=72], hash=1569677858, extras=null, flags=0]]], prepared=0, locked=false, nodeId=a394b7ab-d6b3-4cd8-9b4f-8efef17957a0, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=622803, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=622803, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=622803, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=622803, hasValBytes=false], val=null, startVer=1480614477601, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477601, nodeOrder=72], hash=1593695109, extras=null, flags=0]]], prepared=0, locked=false, nodeId=eb375913-7181-4714-a502-364ff54a1ad1, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=658259, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=658259, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=658259, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=658259, hasValBytes=false], val=null, startVer=1480614477602, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477602, nodeOrder=72], hash=1596638813, extras=null, flags=0]]], prepared=0, locked=false, nodeId=a2ef8fdd-35e4-4731-805e-f0292dba9085, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=711870, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=711870, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=711870, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=711870, hasValBytes=false], val=null, startVer=1480614477603, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477603, nodeOrder=72], hash=1621371162, extras=null, flags=0]]], prepared=0, locked=false, nodeId=89f23db4-4835-479a-b552-4f3b7a2a1264, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=746002, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=746002, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=746002, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=746002, hasValBytes=false], val=null, startVer=1480614477604, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477604, nodeOrder=72], hash=1624282777, extras=null, flags=0]]], prepared=0, locked=false, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=755680, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=755680, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=755680, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=755680, hasValBytes=false], val=null, startVer=1480614477605, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477605, nodeOrder=72], hash=1625182519, extras=null, flags=0]]], prepared=0, locked=false, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=846490, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=846490, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=846490, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=846490, hasValBytes=false], val=null, startVer=1480614477606, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477606, nodeOrder=72], hash=1652916049, extras=null, flags=0]]], prepared=0, locked=false, nodeId=4d743535-0684-42bb-94fa-4d5373f9dcc2, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=911018, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=911018, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=911018, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=911018, hasValBytes=false], val=null, startVer=1480614477607, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477607, nodeOrder=72], hash=1678621598, extras=null, flags=0]]], prepared=0, locked=false, nodeId=94abd5a8-4557-4d97-bdbf-6240a734b8c9, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=961423, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=961423, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=961423, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=961423, hasValBytes=false], val=null, startVer=1480614477608, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477608, nodeOrder=72], hash=1683243073, extras=null, flags=0]]], prepared=0, locked=false, nodeId=89f23db4-4835-479a-b552-4f3b7a2a1264, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=990055, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=990055, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=990055, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=990055, hasValBytes=false], val=null, startVer=1480614477609, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477609, nodeOrder=72], hash=1685980096, extras=null, flags=0]]], prepared=0, locked=false, nodeId=5650c65d-918a-4a27-af67-14d995de536b, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]], IgniteTxKey [key=KeyCacheObjectImpl [val=994840, hasValBytes=false], cacheId=-535738254]=IgniteTxEntry [key=KeyCacheObjectImpl [val=994840, hasValBytes=false], cacheId=-535738254, partId=-1, txKey=IgniteTxKey [key=KeyCacheObjectImpl [val=994840, hasValBytes=false], cacheId=-535738254], val=[op=DELETE, val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[], filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [val=994840, hasValBytes=false], val=null, startVer=1480614477610, ver=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477610, nodeOrder=72], hash=1686106912, extras=null, flags=0]]], prepared=0, locked=false, nodeId=a2ef8fdd-35e4-4731-805e-f0292dba9085, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72]]}], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72], writeVer=null, implicit=false, loc=true, threadId=299, startTime=1480508616516, nodeId=215e4b73-b60c-460b-9154-2949d5de1059, startVer=GridCacheVersion [topVer=91987517, time=1480508616520, order=1480614477545, nodeOrder=72], endVer=null, isolation=READ_COMMITTED, concurrency=PESSIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, preparing=false, invalidParts=null, state=ROLLED_BACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=102, minorTopVer=1], duration=606ms, onePhaseCommit=false], size=52]]], commit=false, mappings=IgniteTxMappingsImpl [], trackable=true, finishOnePhaseCalled=false, innerFuts=[FinishFuture[node=143f09ca-6fab-4c85-aaff-5f023a17ece2, loc=false, done=true], FinishFuture[node=a2ef8fdd-35e4-4731-805e-f0292dba9085, loc=false, done=true], FinishFuture[node=5650c65d-918a-4a27-af67-14d995de536b, loc=false, done=false], FinishFuture[node=adc11bfc-5326-433a-9430-c6bc32c679f9, loc=false, done=true], FinishFuture[node=94619cb7-cb3f-4a60-bc22-2681f933c37a, loc=false, done=false], FinishFuture[node=5640dff9-f585-45e3-bfc5-850365f74bac, loc=false, done=true], FinishFuture[node=b3e4f19e-a359-4649-8040-f0139e0ba1bc, loc=false, done=true], FinishFuture[node=94abd5a8-4557-4d97-bdbf-6240a734b8c9, loc=false, done=true], FinishFuture[node=a394b7ab-d6b3-4cd8-9b4f-8efef17957a0, loc=false, done=false], FinishFuture[node=eb375913-7181-4714-a502-364ff54a1ad1, loc=false, done=true], FinishFuture[node=4f7e0f8b-a895-4971-84e3-bcb80d3fbc24, loc=false, done=false], FinishFuture[node=26199f94-1405-402e-a52e-c8c232d4dc93, loc=false, done=false], FinishFuture[node=4b092c35-1577-48f1-ae3c-7edd4138de60, loc=false, done=true], FinishFuture[node=017cbadc-cc88-4995-b008-18482f75025c, loc=false, done=true], FinishFuture[node=7e96fb15-50ac-47f7-a0cf-6827b808f744, loc=false, done=false], FinishFuture[node=4d743535-0684-42bb-94fa-4d5373f9dcc2, loc=false, done=false], FinishFuture[node=89f23db4-4835-479a-b552-4f3b7a2a1264, loc=false, done=true]], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=o.a.i.i.util.lang.GridFunc$27@738cb650, initFlag=1, lsnrCalls=3, done=false, cancelled=false, err=null, futs=[false, true, false, false, false, true, false, false, false, true, false, false, false, false, false, false, true]]]]class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:786)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:728)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:687)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:157)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:155)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:710)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:102)
at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:673)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
<15:23:37><benchmark-worker-34><yardstick> The benchmark of random operation failed.
java.lang.NullPointerException
at org.apache.ignite.yardstick.IgniteBenchmarkUtils.doInTransaction(IgniteBenchmarkUtils.java:69)
at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeInTransaction(IgniteCacheRandomOperationBenchmark.java:665)
at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.test(IgniteCacheRandomOperationBenchmark.java:152)
at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:176)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> Benchmark driver stops working when one of servers left grid and backupcount 0
> ------------------------------------------------------------------------------
>
> Key: IGNITE-3269
> URL: https://issues.apache.org/jira/browse/IGNITE-3269
> Project: Ignite
> Issue Type: Bug
> Components: yardstick
> Affects Versions: 1.7
> Reporter: Ksenia Rybakova
> Priority: Minor
> Attachments: failed_client_log.zip
>
>
> Test config: 4 hosts with 4 clients and 1 server each, backup count = 0.
> During the test one of the servers was killed. After that one of the clients
> stopped working:
> {noformat}
> [15:31:45,248][WARN ][sys-#27%null%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=fosters-216/10.20.0.216:47100, failureDetectionTimeout=10000]
> [15:31:45,256][WARN ][sys-#27%null%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[fosters-216/10.20.0.216:4
> [15:31:45,276][WARN ][disco-event-worker-#52%null%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=f5bcc4d3-aa56-4e51-8672-e799a42990a6, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:4750
> [15:31:45,324][INFO ][disco-event-worker-#52%null%][GridDiscoveryManager] Topology snapshot [ver=22, servers=3, clients=16, CPUs=64, heap=150.0GB]
> ERROR: The benchmark of random operation failed.
> Type '--help' for usage.
> Finishing main test [ts=1465252305413, date=Mon Jun 06 15:31:45 PDT 2016]
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=9c7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> Type '--help' for usage.
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=6f7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], cache=tx-offheap]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1468)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1972)
> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:914)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.doGet(IgniteCacheRandomOperationBenchmark.java:670)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeRandomOperation(IgniteCacheRandomOperationBenchmark.java:518)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.access$900(IgniteCacheRandomOperationBenchmark.java:83)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark$2.call(IgniteCacheRandomOperationBenchmark.java:628)
> at org.apache.ignite.yardstick.IgniteBenchmarkUtils.doInTransaction(IgniteBenchmarkUtils.java:52)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeInTransaction(IgniteCacheRandomOperationBenchmark.java:619)
> at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.test(IgniteCacheRandomOperationBenchmark.java:153)
> at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:176)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException: Failed to map keys for cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], cache=tx-offheap]
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:660)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:343)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:204)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.access$300(GridPartitionedSingleGetFuture.java:66)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture$4.run(GridPartitionedSingleGetFuture.java:723)
> at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6515)
> at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:803)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ... 1 more
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=558413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=558413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=558413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=558413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=558413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> class org.apache.ignite.IgniteCheckedException: Transaction has been already completed.
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [15:31:45] (err) Failed to execute compound future reducer: GridNearTxFinishFuture [futId=ff7413d2551-758034b3-0248-4d62-b901-be86d119183c, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false, colocatedLocallyMa
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:700)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:644)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:607)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:135)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:133)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:624)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:322)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:246)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:83)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:205)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105)
> at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {noformat}
> Complete log is attached.
> Expected behaviour: client prints some error message in log, but continues performing the test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)