You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Semen Boikov (JIRA)" <ji...@apache.org> on 2016/06/01 06:25:12 UTC
[jira] [Assigned] (IGNITE-3211) "Failed to reinitialize local
partitions", "Failed to wait for completion of partition map exchange"
errors during failover test
[ https://issues.apache.org/jira/browse/IGNITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Semen Boikov reassigned IGNITE-3211:
------------------------------------
Assignee: Semen Boikov
> "Failed to reinitialize local partitions", "Failed to wait for completion of partition map exchange" errors during failover test
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-3211
> URL: https://issues.apache.org/jira/browse/IGNITE-3211
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 1.6
> Reporter: Ksenia Rybakova
> Assignee: Semen Boikov
> Fix For: 1.7
>
>
> "Failed to reinitialize local partitions (preloading will be stopped)" and "Failed to wait for completion of partition map exchange (preloading will not start)" errors occured during failover load test. Complete stack trace see below.
> Load config:
> - 1 client, 20 servers (5 servers per 1 host)
> - warmup 60
> - duration 66h
> - preload 5M
> - key range 10M
> - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT REPLACE
> - backups count 3
> - 3 servers restart every 15 min with 30 sec step, pause between stop and start 5min
> {noformat}
> [08:32:21,002][ERROR][exchange-worker-#83%null%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT]
> class org.apache.ignite.IgniteException: null
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
> at java.lang.String.valueOf(String.java:2849)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteException: null
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
> at java.lang.String.valueOf(String.java:2849)
> at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> ... 15 more
> Caused by: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
> at java.util.AbstractMap.toString(AbstractMap.java:518)
> at java.lang.String.valueOf(String.java:2849)
> at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> ... 20 more
> [08:32:21,072][ERROR][exchange-worker-#83%null%][GridCachePartitionExchangeManager] Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, isClient=false], topVer=39, nodeId8=20ddc8b7, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1464363130370]], crd=TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1464363130370, endTime=1464363141057, ignoreInterrupts=false, state=DONE], init=false, topSnapshot=null, lastVer=GridCacheVersion [topVer=75840698, time=1464363140466, order=1464462577173, nodeOrder=38], partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=2, done=false, cancelled=false, err=null, futs=[true, false, true]], affChangeMsg=CacheAffinityChangeMessage [id=9560ad2f451-621c2001-9d60-40f4-ab21-20834a2b6b33, topVer=AffinityTopologyVersion [topVer=39, minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], skipPreload=false, clientOnlyExchange=false, initTs=1464363130370, centralizedAff=false, evtLatch=0, remaining=[e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4, a550929c-1f4b-4de3-8d57-47b4a8232010, c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, fc79bff1-480f-44b3-9915-2548f937726c, c013837c-b46e-4667-9e47-e3c41412b5a1, ba89970d-8908-4276-9d3c-b679a335c2d2, a7f0feb2-b932-4843-ac93-560fc547700e, 3c462bba-1d12-43ab-ad21-761f9ef94aa1, 9c184df6-e5e0-49b3-9836-65932d58be6a, 317b61fc-186b-4bf7-9622-6c4094409814, 425b461c-5117-403d-af2b-e323b7c1aa39, a4d0911c-d54a-4a36-b911-b74eddc3f0cc, d0e84363-590d-45bf-bad3-b88cde5d03fe, b51cc458-c884-41ce-9e5b-b83ea111218b, 6970f13f-fe82-4583-a128-af93f86e99e7, a9b8fe06-cda9-4907-871d-0684bcc2c3ca, c978e55f-4de6-4ffd-98b1-459215db2232, cab5d0e0-7365-4774-8f99-d9f131c5d896, 0859feda-b70d-4fab-870a-4db18bd5984b], srvNodes=[TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=cab5d0e0-7365-4774-8f99-d9f131c5d896, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500, /10.20.0.221:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1464360659615, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500, /10.20.0.222:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1464360660162, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=0859feda-b70d-4fab-870a-4db18bd5984b, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500, /10.20.0.223:47500, /127.0.0.1:47500], discPort=47500, order=4, intOrder=4, lastExchangeTime=1464360660689, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=ba89970d-8908-4276-9d3c-b679a335c2d2, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501, /10.20.0.216:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5, lastExchangeTime=1464360661276, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=317b61fc-186b-4bf7-9622-6c4094409814, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501, /10.20.0.221:47501, /127.0.0.1:47501], discPort=47501, order=6, intOrder=6, lastExchangeTime=1464360662086, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=425b461c-5117-403d-af2b-e323b7c1aa39, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501, /10.20.0.222:47501, /127.0.0.1:47501], discPort=47501, order=7, intOrder=7, lastExchangeTime=1464360662471, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=a550929c-1f4b-4de3-8d57-47b4a8232010, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501, /10.20.0.223:47501, /127.0.0.1:47501], discPort=47501, order=8, intOrder=8, lastExchangeTime=1464360662886, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47502, /10.20.0.216:47502, /127.0.0.1:47502], discPort=47502, order=9, intOrder=9, lastExchangeTime=1464360663373, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=b51cc458-c884-41ce-9e5b-b83ea111218b, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502, /10.20.0.221:47502, /127.0.0.1:47502], discPort=47502, order=10, intOrder=10, lastExchangeTime=1464360664058, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=fc79bff1-480f-44b3-9915-2548f937726c, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502, /10.20.0.222:47502, /127.0.0.1:47502], discPort=47502, order=11, intOrder=11, lastExchangeTime=1464360664685, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=a7f0feb2-b932-4843-ac93-560fc547700e, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502, /10.20.0.223:47502, /127.0.0.1:47502], discPort=47502, order=12, intOrder=12, lastExchangeTime=1464360665216, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=9c184df6-e5e0-49b3-9836-65932d58be6a, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47503, /10.20.0.216:47503, /127.0.0.1:47503], discPort=47503, order=13, intOrder=13, lastExchangeTime=1464360665661, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=6970f13f-fe82-4583-a128-af93f86e99e7, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47504, /10.20.0.216:47504, /127.0.0.1:47504], discPort=47504, order=17, intOrder=17, lastExchangeTime=1464360668158, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=d0e84363-590d-45bf-bad3-b88cde5d03fe, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47504, /10.20.0.221:47504, /127.0.0.1:47504], discPort=47504, order=18, intOrder=18, lastExchangeTime=1464360668768, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=c013837c-b46e-4667-9e47-e3c41412b5a1, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47504, /10.20.0.222:47504, /127.0.0.1:47504], discPort=47504, order=19, intOrder=19, lastExchangeTime=1464360669745, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=3c462bba-1d12-43ab-ad21-761f9ef94aa1, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47504, /10.20.0.223:47504, /127.0.0.1:47504], discPort=47504, order=20, intOrder=20, lastExchangeTime=1464360669836, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=c978e55f-4de6-4ffd-98b1-459215db2232, addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47503, /10.20.0.221:47503, /127.0.0.1:47503], discPort=47503, order=37, intOrder=28, lastExchangeTime=1464362949633, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=a9b8fe06-cda9-4907-871d-0684bcc2c3ca, addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47503, /10.20.0.222:47503, /127.0.0.1:47503], discPort=47503, order=38, intOrder=29, lastExchangeTime=1464362981162, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false], TcpDiscoveryNode [id=a4d0911c-d54a-4a36-b911-b74eddc3f0cc, addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47503, /10.20.0.223:47503, /127.0.0.1:47503], discPort=47503, order=39, intOrder=30, lastExchangeTime=1464363011016, loc=false, ver=1.6.0#20160525-sha1:48321a40, isClient=false]], super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: null, startTime=1464363130370, endTime=1464363141003, ignoreInterrupts=false, state=DONE]]
> class org.apache.ignite.IgniteCheckedException: null
> at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7067)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:168)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1335)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
> at java.util.AbstractMap.toString(AbstractMap.java:518)
> at java.lang.String.valueOf(String.java:2849)
> at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
> at java.lang.String.valueOf(String.java:2849)
> at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
> at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
> at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
> at java.lang.String.valueOf(String.java:2849)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)