You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Alexey Goncharuk (JIRA)" <ji...@apache.org> on 2018/10/29 10:13:02 UTC
[jira] [Resolved] (IGNITE-10013) Node restart may lead to NPE in
GridDhtPartitionsExchangeFuture
[ https://issues.apache.org/jira/browse/IGNITE-10013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Goncharuk resolved IGNITE-10013.
---------------------------------------
Resolution: Duplicate
> Node restart may lead to NPE in GridDhtPartitionsExchangeFuture
> ---------------------------------------------------------------
>
> Key: IGNITE-10013
> URL: https://issues.apache.org/jira/browse/IGNITE-10013
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.7
> Reporter: Sergey Kozlov
> Assignee: Pavel Kovalenko
> Priority: Critical
> Fix For: 2.7
>
>
> 1. Start 4 nodes with ~100 caches, create a few dynamic caches
> 2. Load data (~500 entries per cache)
> 3. Restart every node starting from first
> The first node may throw NPE:
> {noformat}
> [20:00:18,355][INFO][exchange-worker-#63][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=c7a332ee-ee85-4604-aa93-919f15e34e36, crdVer=1540497597005, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=cc8ed618-2f14-46cf-be03-645291c397aa, customEvt=null, allowMerge=true]
> [20:00:18,358][INFO][exchange-worker-#63][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null]
> [20:00:18,364][SEVERE][sys-#77][GridCacheIoManager] Failed processing message [senderId=c50165fb-1afd-4b3e-8fba-ac825aa532fa, msg=GridDhtPartitionsSingleMessage [parts=HashMap {-553318226=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553318133=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553318412=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315529=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315497=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315498=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=100], -553318319=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], 374280891=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=32], 374280890=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=64], 374280889=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=128], 374280888=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], 374280887=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=23], 374280886=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=31], 374280885=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=61], 374280884=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=490], -553315527=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553315495=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=5, size=512], -553315528=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=490], -553315496=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=512], -553315526=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=4, minorTopVer=1], updateSeq=4, size=512]}, partCntrs=null, partsSizes=HashMap {-553318226=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553318133=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553318412=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315529=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315497=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315498=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553318319=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, 374280891=HashMap {0=18, 1=24, 2=24, 3=24, 4=24, 5=18, 6=18, 7=18, 8=18, 9=18, 10=18, 11=18, 12=18, 13=18, 14=18, 15=18, 16=18, 17=18, 18=18, 19=18, 20=18, 21=18, 22=18, 23=18, 24=18, 25=18, 26=18, 27=18, 28=18, 29=18, 30=18, 31=18}, 374280890=HashMap {0=26, 1=52, 2=52, 3=52, 4=52, 5=52, 6=52, 7=52, 8=52, 9=52, 10=52, 11=52, 12=52, 13=52, 14=52, 15=52, 16=52, 17=52, 18=52, 19=52, 20=52, 21=52, 22=52, 23=52, 24=52, 25=52, 26=52, 27=52, 28=52, 29=52, 30=52, 31=52, 32=52, 33=52, 34=52, 35=52, 36=52, 37=26, 38=26, 39=26, 40=26, 41=26, 42=26, 43=26, 44=26, 45=26, 46=26, 47=26, 48=26, 49=26, 50=26, 51=26, 52=26, 53=26, 54=26, 55=26, 56=26, 57=26, 58=26, 59=26, 60=26, 61=26, 62=26, 63=26}, 374280889=HashMap {1=14, 2=14, 3=14, 4=14, 5=14, 6=14, 7=14, 8=14, 9=14, 10=14, 11=14, 12=14, 13=14, 14=14, 15=14, 16=14, 17=14, 18=14, 19=14, 20=14, 21=14, 22=14, 23=14, 24=14, 25=14, 26=14, 27=14, 28=14, 29=14, 30=14, 31=14, 32=14, 33=14, 34=14, 35=14, 36=14, 37=14, 38=14, 39=14, 40=14, 41=14, 42=14, 43=14, 44=14, 45=14, 46=14, 47=14, 48=14, 49=14, 50=14, 51=14, 52=14, 53=14, 54=14, 55=14, 56=14, 57=14, 58=14, 59=14, 60=14, 61=14, 62=14, 63=14, 64=14, 65=14, 66=14, 67=14, 68=14, 69=14, 70=14, 71=14, 72=14, 73=14, 74=14, 75=14, 76=14, 77=14, 78=14, 79=14, 80=14, 81=14, 82=14, 83=14, 84=14, 85=14, 86=14, 87=14, 88=14, 89=14, 90=14, 91=14, 92=14, 93=14, 94=14, 95=14, 96=14, 97=14, 98=14, 99=14, 100=14}, 374280888=HashMap {1=12, 2=12, 3=12, 4=12, 5=12, 6=12, 7=12, 8=12, 9=12, 10=12, 11=12, 12=12, 13=12, 14=12, 15=12, 16=12, 17=12, 18=12, 19=12, 20=12, 21=12, 22=12, 23=12, 24=12, 25=12, 26=12, 27=12, 28=12, 29=12, 30=12, 31=12, 32=12, 33=12, 34=12, 35=12, 36=12, 37=12, 38=12, 39=12, 40=12, 41=12, 42=12, 43=12, 44=12, 45=12, 46=12, 47=12, 48=12, 49=12, 50=12, 51=12, 52=12, 53=12, 54=12, 55=12, 56=12, 57=12, 58=12, 59=12, 60=12, 61=12, 62=12, 63=12, 64=12, 65=12, 66=12, 67=12, 68=12, 69=12, 70=12, 71=12, 72=12, 73=12, 74=12, 75=12, 76=12, 77=12, 78=12, 79=12, 80=12, 81=12, 82=12, 83=12, 84=12, 85=12, 86=12, 87=12, 88=12, 89=12, 90=12, 91=12, 92=12, 93=12, 94=12, 95=12, 96=12, 97=12, 98=12, 99=12, 100=12}, 374280887=HashMap {0=18, 1=24, 2=24, 3=24, 4=24, 5=18, 6=18, 7=18, 9=18, 11=18, 13=18, 15=18, 16=18, 19=18, 20=18, 21=18, 22=18, 24=18, 25=18, 27=18, 28=18, 30=18, 31=18}, 374280886=HashMap {0=26, 1=52, 2=52, 3=52, 4=52, 5=52, 6=52, 7=52, 9=52, 11=52, 13=52, 20=52, 21=52, 22=52, 24=52, 27=52, 28=52, 31=52, 32=52, 33=52, 34=52, 35=52, 36=52, 39=26, 42=26, 43=26, 47=26, 55=26, 57=26, 58=26, 63=26}, 374280885=HashMap {1=14, 2=14, 3=14, 4=14, 5=14, 6=14, 7=14, 9=14, 11=14, 13=14, ... and 3711 skipped ...75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553315528=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 9=1, 11=1, 13=1, 20=1, 21=1, 22=1, 24=1, 27=1, 28=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 39=1, 42=1, 43=1, 47=1, 55=1, 57=1, 58=1, 63=1, 64=1, 65=1, 66=1, 67=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 79=1, 80=1, 82=1, 84=1, 86=1, 88=1, 90=1, 95=1, 96=1, 99=1}, -553315496=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}, -553315526=HashMap {1=1, 2=1, 3=1, 4=1, 5=1, 6=1, 7=1, 8=1, 9=1, 10=1, 11=1, 12=1, 13=1, 14=1, 15=1, 16=1, 17=1, 18=1, 19=1, 20=1, 21=1, 22=1, 23=1, 24=1, 25=1, 26=1, 27=1, 28=1, 29=1, 30=1, 31=1, 32=1, 33=1, 34=1, 35=1, 36=1, 37=1, 38=1, 39=1, 40=1, 41=1, 42=1, 43=1, 44=1, 45=1, 46=1, 47=1, 48=1, 49=1, 50=1, 51=1, 52=1, 53=1, 54=1, 55=1, 56=1, 57=1, 58=1, 59=1, 60=1, 61=1, 62=1, 63=1, 64=1, 65=1, 66=1, 67=1, 68=1, 69=1, 70=1, 71=1, 72=1, 73=1, 74=1, 75=1, 76=1, 77=1, 78=1, 79=1, 80=1, 81=1, 82=1, 83=1, 84=1, 85=1, 86=1, 87=1, 88=1, 89=1, 90=1, 91=1, 92=1, 93=1, 94=1, 95=1, 96=1, 97=1, 98=1, 99=1, 100=1}}, partHistCntrs=null, err=null, client=false, compress=false, finishMsg=null, activeQryTrackers=null, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=GridCacheVersion [topVer=0, order=1540497610955, nodeOrder=0], super=GridCacheMessage [msgId=19, depInfo=null, err=null, skipPrepare=false]]]]
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.partitionChangesInProgress(GridDhtPartitionsExchangeFuture.java:4577)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.exchangeInProgress(GridCachePartitionExchangeManager.java:2307)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1000(GridCachePartitionExchangeManager.java:142)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:350)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:333)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3102)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3081)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [20:00:18,365][INFO][exchange-worker-#63][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=c7a332ee-ee85-4604-aa93-919f15e34e36, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=cc8ed618-2f14-46cf-be03-645291c397aa, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=5, intOrder=5, lastExchangeTime=1540497616253, loc=false, ver=2.7.1#20181023-sha1:b29a1bd9, isClient=true], done=true], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], durationFromInit=23]
> [20:00:18,372][SEVERE][sys-#77][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.partitionChangesInProgress(GridDhtPartitionsExchangeFuture.java:4577)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.exchangeInProgress(GridCachePartitionExchangeManager.java:2307)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1000(GridCachePartitionExchangeManager.java:142)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:350)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:333)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3102)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3081)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [20:00:18,373][WARNING][sys-#77][FailureProcessor] No deadlocked threads detected.
> [20:00:18,527][INFO][exchange-worker-#63][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=true]
> [20:00:18,530][WARNING][sys-#77][FailureProcessor] Thread dump at 2018/10/25 20:00:18 UTC
> Thread [name="srvc-deploy-#88", id=129, state=WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7769e8eb, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#87", id=128, state=TIMED_WAITING, blockCnt=1, waitCnt=17]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#86", id=127, state=TIMED_WAITING, blockCnt=5, waitCnt=13]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#85", id=126, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#84", id=125, state=TIMED_WAITING, blockCnt=0, waitCnt=22]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#83", id=124, state=TIMED_WAITING, blockCnt=0, waitCnt=15]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#81", id=123, state=TIMED_WAITING, blockCnt=0, waitCnt=18]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#82", id=122, state=TIMED_WAITING, blockCnt=0, waitCnt=19]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#80", id=121, state=TIMED_WAITING, blockCnt=0, waitCnt=6]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#79", id=120, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#78", id=119, state=TIMED_WAITING, blockCnt=0, waitCnt=18]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#77", id=118, state=RUNNABLE, blockCnt=2, waitCnt=11]
> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
> at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1364)
> at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:132)
> - locked o.a.i.i.processors.failure.FailureProcessor@26c61451
> at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:108)
> at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1071)
> at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@7ba7092f
> Thread [name="Thread-7", id=117, state=TIMED_WAITING, blockCnt=0, waitCnt=306]
> Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Thread-5", id=116, state=TIMED_WAITING, blockCnt=0, waitCnt=635]
> Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Thread-6", id=115, state=TIMED_WAITING, blockCnt=0, waitCnt=643]
> Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#76", id=114, state=TIMED_WAITING, blockCnt=0, waitCnt=12]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#75", id=113, state=WAITING, blockCnt=2, waitCnt=625]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#74", id=112, state=WAITING, blockCnt=5, waitCnt=606]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#73", id=111, state=WAITING, blockCnt=1, waitCnt=617]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#72", id=110, state=WAITING, blockCnt=5, waitCnt=595]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14a5a8d6, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="db-checkpoint-thread-#71", id=109, state=RUNNABLE, blockCnt=0, waitCnt=5]
> at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
> at sun.nio.ch.SimpleAsynchronousFileChannelImpl.force(SimpleAsynchronousFileChannelImpl.java:162)
> at o.a.i.i.processors.cache.persistence.file.AsyncFileIO.force(AsyncFileIO.java:183)
> at o.a.i.i.processors.cache.persistence.file.AsyncFileIO.force(AsyncFileIO.java:178)
> at o.a.i.i.processors.cache.persistence.file.FilePageStore.sync(FilePageStore.java:638)
> at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3306)
> at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3096)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@40f5ed36
> Thread [name="sys-#70", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#69", id=107, state=TIMED_WAITING, blockCnt=0, waitCnt=21]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="ttl-cleanup-worker-#68", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
> at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:149)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#67", id=105, state=TIMED_WAITING, blockCnt=1, waitCnt=21]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Thread-4", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=18477]
> Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="wal-segment-syncer-#66", id=103, state=TIMED_WAITING, blockCnt=0, waitCnt=12]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
> at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogManager.java:3514)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="wal-file-archiver%null-#65", id=102, state=WAITING, blockCnt=1, waitCnt=2]
> Lock [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage@56871de5, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.awaitSegment(SegmentCurrentStateStorage.java:73)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.waitNextSegmentForArchivation(SegmentCurrentStateStorage.java:90)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.waitNextSegmentForArchivation(SegmentAware.java:79)
> at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1719)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="tcp-disco-sock-reader-#8", id=101, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked java.io.BufferedInputStream@6319d7c5
> at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
> at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2653)
> at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2669)
> at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
> at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
> at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
> at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
> at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10085)
> at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="sys-#64", id=100, state=TIMED_WAITING, blockCnt=2, waitCnt=20]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3bddfb37, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Keep-Alive-Timer", id=97, state=TIMED_WAITING, blockCnt=0, waitCnt=4]
> at java.lang.Thread.sleep(Native Method)
> at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="session-timeout-worker-#54", id=79, state=TIMED_WAITING, blockCnt=0, waitCnt=22]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="exchange-worker-#63", id=92, state=RUNNABLE, blockCnt=2, waitCnt=23214]
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionMap.<init>(GridDhtPartitionMap.java:96)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap.<init>(GridDhtPartitionFullMap.java:86)
> at o.a.i.i.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.partitionMap(GridDhtPartitionTopologyImpl.java:1326)
> at o.a.i.i.processors.cache.GridCachePartitionExchangeManager.createPartitionsFullMessage(GridCachePartitionExchangeManager.java:1175)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createPartitionsMessage(GridDhtPartitionsExchangeFuture.java:1900)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$700(GridDhtPartitionsExchangeFuture.java:139)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$3.apply(GridDhtPartitionsExchangeFuture.java:2688)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$3.apply(GridDhtPartitionsExchangeFuture.java:2653)
> at o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
> at o.a.i.i.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitAndReplyToNode(GridDhtPartitionsExchangeFuture.java:2653)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:2708)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:139)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2612)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2600)
> at o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
> at o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
> at o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:497)
> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:476)
> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2195)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:138)
> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:453)
> at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:823)
> at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
> at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="ignite-update-notifier-timer", id=91, state=TIMED_WAITING, blockCnt=1, waitCnt=3]
> Lock [object=java.util.TaskQueue@5ac33068, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> at java.util.TimerThread.run(Timer.java:505)
> Thread [name="upd-ver-checker", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=5]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="disco-event-worker-#62", id=89, state=WAITING, blockCnt=0, waitCnt=18]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35b0ea5b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="tcp-disco-srvr-#3", id=88, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="tcp-disco-msg-worker-#2", id=87, state=TIMED_WAITING, blockCnt=0, waitCnt=2074]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bb6b13d, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
> at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
> at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183)
> at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="disco-notifier-worker-#61", id=86, state=WAITING, blockCnt=0, waitCnt=20]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@763ce271, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-data-loader-flusher-#60", id=85, state=WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35bfdcae, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
> at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-3-#58", id=83, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7ef1a70
> - locked java.util.Collections$UnmodifiableSet@693e7ce9
> - locked sun.nio.ch.EPollSelectorImpl@788024d5
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-2-#57", id=82, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3391ddc7
> - locked java.util.Collections$UnmodifiableSet@4cd27d5f
> - locked sun.nio.ch.EPollSelectorImpl@56119918
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-1-#56", id=81, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@2fceb983
> - locked java.util.Collections$UnmodifiableSet@120c2658
> - locked sun.nio.ch.EPollSelectorImpl@4deb89c3
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-0-#55", id=80, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17e8003f
> - locked java.util.Collections$UnmodifiableSet@4db0b1e0
> - locked sun.nio.ch.EPollSelectorImpl@3e7d70e8
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-tcp-rest-#59", id=84, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@2fc94375
> - locked java.util.Collections$UnmodifiableSet@78f7574b
> - locked sun.nio.ch.EPollSelectorImpl@3080c604
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-3-#52", id=77, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@65af1acc
> - locked java.util.Collections$UnmodifiableSet@182fe7
> - locked sun.nio.ch.EPollSelectorImpl@42176da0
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-2-#51", id=76, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@74113300
> - locked java.util.Collections$UnmodifiableSet@69752e34
> - locked sun.nio.ch.EPollSelectorImpl@28d343ea
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-1-#50", id=75, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5150513d
> - locked java.util.Collections$UnmodifiableSet@2ca4f124
> - locked sun.nio.ch.EPollSelectorImpl@6ce06e0a
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-0-#49", id=74, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5e9eac6f
> - locked java.util.Collections$UnmodifiableSet@d14c738
> - locked sun.nio.ch.EPollSelectorImpl@6b23a6f
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-client-listener-#53", id=78, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@69cd848d
> - locked java.util.Collections$UnmodifiableSet@429ba00f
> - locked sun.nio.ch.EPollSelectorImpl@691d9263
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Thread-1", id=73, state=TIMED_WAITING, blockCnt=0, waitCnt=7722]
> Lock [object=java.util.concurrent.SynchronousQueue$TransferStack@2cb3331b, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="tcp-comm-worker-#1", id=72, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e6bbc6d, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="grid-nio-worker-tcp-comm-7-#47", id=70, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7b87b6c1
> - locked java.util.Collections$UnmodifiableSet@6403fa66
> - locked sun.nio.ch.EPollSelectorImpl@6108f20f
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-6-#46", id=69, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@42a3b297
> - locked java.util.Collections$UnmodifiableSet@4d425bc
> - locked sun.nio.ch.EPollSelectorImpl@7f7f2f8a
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-5-#45", id=68, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@594122e0
> - locked java.util.Collections$UnmodifiableSet@6a91455
> - locked sun.nio.ch.EPollSelectorImpl@237538f9
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-4-#44", id=67, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@25705062
> - locked java.util.Collections$UnmodifiableSet@7576300e
> - locked sun.nio.ch.EPollSelectorImpl@3bb39949
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-3-#43", id=66, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@310bdc36
> - locked java.util.Collections$UnmodifiableSet@16df6935
> - locked sun.nio.ch.EPollSelectorImpl@526ddda0
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-2-#42", id=65, state=RUNNABLE, blockCnt=9, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@75b37f75
> - locked java.util.Collections$UnmodifiableSet@1a36e799
> - locked sun.nio.ch.EPollSelectorImpl@774da32b
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-1-#41", id=64, state=RUNNABLE, blockCnt=7, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4bf1f0bf
> - locked java.util.Collections$UnmodifiableSet@25d76663
> - locked sun.nio.ch.EPollSelectorImpl@8f0476f
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-0-#40", id=63, state=RUNNABLE, blockCnt=4, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@12bee9a1
> - locked java.util.Collections$UnmodifiableSet@560e9741
> - locked sun.nio.ch.EPollSelectorImpl@4874179d
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-tcp-comm-#48", id=71, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@4235eac3
> - locked java.util.Collections$UnmodifiableSet@2723e0c5
> - locked sun.nio.ch.EPollSelectorImpl@535cfff
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-timeout-worker-#39", id=62, state=TIMED_WAITING, blockCnt=2, waitCnt=59]
> Lock [object=java.lang.Object@6d15e83b, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="ignite-clock", id=61, state=TIMED_WAITING, blockCnt=0, waitCnt=2247]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3379)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#38", id=60, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#37", id=59, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#36", id=58, state=TIMED_WAITING, blockCnt=1, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#35", id=57, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#34", id=56, state=TIMED_WAITING, blockCnt=1, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#33", id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b74a8, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="jvm-pause-detector-worker", id=54, state=TIMED_WAITING, blockCnt=0, waitCnt=459]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)
> Thread [name="data-streamer-stripe-15-#32", id=53, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-14-#31", id=52, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-13-#30", id=51, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-12-#29", id=50, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-11-#28", id=49, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-10-#27", id=48, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-9-#26", id=47, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-8-#25", id=46, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-7-#24", id=45, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-6-#23", id=44, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-5-#22", id=43, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-4-#21", id=42, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-3-#20", id=41, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-2-#19", id=40, state=WAITING, blockCnt=1, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-1-#18", id=39, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-0-#17", id=38, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-15-#16", id=37, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-14-#15", id=36, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-13-#14", id=35, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-12-#13", id=34, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-11-#12", id=33, state=WAITING, blockCnt=1, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-10-#11", id=32, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-9-#10", id=31, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-8-#9", id=30, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-7-#8", id=29, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-6-#7", id=28, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-5-#6", id=27, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-4-#5", id=26, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-3-#4", id=25, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-2-#3", id=24, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-1-#2", id=23, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-0-#1", id=22, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-0", id=21, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-62446", id=20, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-0", id=19, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]
> Thread [name="Finalizer", id=3, state=WAITING, blockCnt=87, waitCnt=5]
> Lock [object=java.lang.ref.ReferenceQueue$Lock@380699a6, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=4, waitCnt=4]
> Lock [object=java.lang.ref.Reference$Lock@1367e36e, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> Thread [name="main", id=1, state=WAITING, blockCnt=0, waitCnt=34]
> Lock [object=java.util.concurrent.CountDownLatch$Sync@51f18509, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334)
> [20:00:18,531][SEVERE][sys-#77][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)